Systems, methods and computer program products for scalable, low-latency processing of streaming data

ABSTRACT

A system for processing network flow monitoring data includes a data collection input coupled to at least one incoming data stream; a load balancing operator is operable to distribute the network monitoring messages amongst a plurality of downstream processing units; a plurality of collector processing sub-units downstream from the load balancing operator are operable to generate network flow records from the received network monitoring messages in a first format; a plurality of parser sub-units coupled to the plurality of collector processing sub-units are operable to generate parsed network flow records by converting the received network flow records from the first format to a second format; and a combiner operator coupled to the plurality of parser sub-units is operable to combine the parsed network flow records from the plurality of parser sub-units into a synchronous output data stream.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from U.S. Provisional PatentApplication Ser. No. 62/724,789, filed Aug. 30, 2018, the entirecontents of which are hereby incorporated by reference.

FIELD

The described embodiments relate to data processing systems, and inparticular to systems, methods and computer program products forcollecting and processing large volumes of streaming data.

BACKGROUND

The following is not an admission that anything discussed below is partof the prior art or part of the common general knowledge of a personskilled in the art.

Electronic communications networks are a crucial part of the modernworld. These communication networks enable individuals to access anddisseminate large volumes of data/information. This data can includeimportant, confidential and sensitive data for individuals orcorporations. Accessing data using communication networks has become aroutine and essential aspect of organizational operations.

As the world is becoming increasingly digitized, the volume of datashared and stored using electronic communication networks is increasing.The types and volume of sensitive data that is shared or accessible viathese communication networks is also increasing. As a result, it isimportant to ensure that these communication networks remainoperational. Additionally, it can be important to ensure thatunauthorized access to communication networks can be reduced orprevented so that sensitive data can be protected.

In order to protect communication networks against issues such as faultsor security breaches, a variety of network monitoring applications canbe used. These applications gather information about the network andanalyze the received data to detect events of interest such as faults,anomalies and intrusions. However, in order for these applications tooperate successfully, the information must be provided in a formatsuitable for analysis. Providing the appropriate data for monitoring andanalysis can be difficult, particularly when a large volume of data iscollected. As more and more applications become reliant on communicationnetworks, particularly with the emergence and growth of Internet ofThings devices and applications, the volume and diversity of datagenerated within these networks will continue to increase dramatically.

SUMMARY

The following introduction is provided to introduce the reader to themore detailed discussion to follow. The introduction is not intended tolimit or define any claimed or as yet unclaimed invention. One or moreinventions may reside in any combination or sub-combination of theelements or process steps disclosed in any part of this documentincluding its claims and figures.

In a first broad aspect, there is provided a system for processingstreaming data. The system includes a data collection input coupled toat least one incoming data stream; a plurality of data processingsub-units, where each data processing sub-unit has a data input and adata output, and each data processing sub-unit is operable to receive aninput data stream at the data input, modify data in the input datastream, and output an outgoing data stream that includes the modifieddata via the data output; a plurality of operators connecting the datacollection input, the plurality of data processing sub-units, and atleast one data output unit in a directed acyclic graph in which dataflows from the data collection input through the plurality of dataprocessing sub-units to the at least one data output unit; where, for atleast some first data processing units, the data input of each dataprocessing sub-unit in the at least some first data processing units isdirectly connected to a sub-unit input operator that transmits datausing the User Datagram Protocol (UDP) and for at least some second dataprocessing units the data output of each data processing sub-unit in theat least some second data processing units is directly connected to asub-unit output operator that receives data using UDP.

In some embodiments, the system also includes a system manager coupledto the plurality of data processing sub-units and each data processingsub-unit includes a data buffer coupled to the data input where thesystem manager is configured to: monitor buffer usage of each databuffer; determine that buffer usage for a particular data buffer hasreached a usage threshold; and modify the acyclic graph to reduce thebuffer usage of the particular data buffer in response to the bufferusage reaching the usage threshold.

In some embodiments, the plurality of data processing sub-units includesat least one plurality of parallel sub-unit instances, each plurality ofparallel sub-unit instances including at least two data processingsub-units each of which are operable to modify the data in the inputdata stream in the same manner and each data processing sub-unit in theat least two data processing sub-units is connected to the samepreceding data processing sub-unit in the directed acyclic graph.

In some embodiments, the data output of the preceding data processingsub-unit is connected to an output duplicating operator that duplicatesthe outgoing data stream of the preceding data processing sub-unit usingUDP multicast.

In some embodiments, the data output of the preceding data processingsub-unit is connected to a distributive operator that distributes theoutgoing data stream of the preceding data processing sub-unit amongstthe at least two data processing sub-units.

In some embodiments, the system also includes a system manager coupledto the at least one data output unit, the plurality of data processingsub-units and the plurality of operators, where the system manager isconfigured to: receive a processing request from the at least one dataoutput unit; determine whether the directed acyclic graph is configuredto satisfy the processing request; and upon determining that thedirected acyclic graph is not configured to satisfy the processingrequest, modify the directed acyclic graph to enable the directedacyclic graph to satisfy the processing request.

In some embodiments, the system manager is configured to modify thedirected acyclic graph while the system is operational.

In some embodiments, modifying the directed acyclic graph includesinserting at least one additional instance of one of the data processingsub-units into the directed acyclic graph.

In some embodiments, modifying the directed acyclic graph includesinserting at least one new data processing sub-unit into the directedacyclic graph.

In some embodiments, modifying the directed acyclic graph includesmodifying at least one of the operators positioned between the pluralityof data processing sub-units.

In some embodiments, the system also includes a system manager coupledto the at least one data output unit, the plurality of data processingsub-units and the plurality of operators, where the system manager isconfigured to limit the volume of data received at the data collectioninput from the at least one incoming data stream data that istransmitted to the directed acyclic graph.

In some embodiments, the system manager is further configured to:monitor the throughput of each of the data processing sub-units; andadjust the volume of data that is transmitted to the directed acyclicgraph based on the monitored throughput.

In some embodiments, the system also includes a system manager coupledto the at least one data analysis application, the plurality of dataprocessing sub-units and the plurality of operators, where the systemmanager is configured to: monitor the performance of each of the dataprocessing sub-units; identify a performance deficiency in the monitoredperformance of at least one of the data processing sub-units; and modifythe directed acyclic graph in response to the identified performancedeficiency.

In some embodiments, the directed acyclic graph is defined as aplurality of data processing segments; where each data processingsegment includes a sequence input operator, a data processing sequencethat includes one or more data processing sub-units directly connectedin sequence, and a sequence output operator; and an upstream end of thedata processing sequence is connected to the sequence input operator anda downstream end of the data processing sequence is connected to thesequence output operator.

In some embodiments, the system also includes a particular plurality ofparallel data processing segments, where each parallel data processingsegment in the particular plurality of parallel data processing segmentsincludes an instance of the same sequence input operator, an instance ofthe same data processing sequence, and an instance of the same sequenceoutput operator.

In some embodiments, the at least one data output unit may include atleast one of a data analysis application and a real-time storageapplication.

In some embodiments, the system may include a compression sub-unitupstream from the real-time storage application.

In some embodiments, the real-time storage application may be configuredto store output data with time index data determined from the parsednetwork flow records.

In some embodiments the system may include a sequence of at least oneparser sub-unit, at least one enrichment sub-unit and at least onecompression, and the real-time storage application may be configured tostore output data from the sequence of the at least one parser sub-unit,at least one enrichment sub-unit and at least one compression with timeindex data for subsequent retrieval. In some embodiments, the time indexdata is determined by the at least one parser sub-unit.

In accordance with an aspect of this disclosure there is provided amethod for processing streaming data, the method including: receiving atleast one incoming data stream; modifying the data in the incoming datastream using a plurality of data processing sub-units, where each dataprocessing sub-unit is operable to receive an input data stream, modifydata in the input data stream, and output an outgoing data stream thatincludes the modified data; routing the data from the incoming datastream through a directed acyclic graph to at least one data outputunit, where the directed acyclic graph includes the plurality of dataprocessing sub-units and a plurality of operators connecting the datacollection input, the plurality of data processing sub-units, and the atleast one data output unit, and the operators are operable to route thedata through the directed acyclic graph; where routing the data throughthe incoming data stream includes, for at least some first dataprocessing units, transmitting data to each data processing sub-unit inthe at least some first data processing units using the User DatagramProtocol (UDP) and for at least some second data processing unitstransmitting data from each data processing sub-unit in the at leastsome second data processing units using UDP.

In some embodiments, the method also includes buffering the input datareceived at each of the data processing sub-units using a correspondingdata buffer; monitoring the buffer usage of each data buffer;determining that buffer usage for a particular data buffer has reached ausage threshold; and modifying the acyclic graph to reduce the bufferusage of the particular data buffer in response to the buffer usagereaching the usage threshold.

In some embodiments, routing the data through the directed acyclic graphincludes, for at least one data processing sub-unit, duplicating theoutgoing data stream using UDP multicast and transmitting the duplicatedoutgoing data stream to a plurality of parallel sub-unit instances. Insome embodiments, each sub-unit instance in the plurality of parallelsub-unit instances modifies the data in the outgoing data stream in thesame manner. In other embodiments, some of the sub-unit instances in theplurality of parallel sub-unit instances may modify the data in theoutgoing data stream in different ways.

In some embodiments, routing the data through the directed acyclic graphincludes, for at least one data processing sub-unit, distributing theoutgoing data stream amongst a plurality of parallel sub-unit instances,where each sub-unit instances in the plurality of parallel sub-unitinstances modifies the data in the outgoing data stream in the samemanner.

In some embodiments, the method also includes: receiving a processingrequest from the at least one data output unit; determining that thedirected acyclic graph is not configured to satisfy the processingrequest; and upon determining that the directed acyclic graph is notconfigured to satisfy the processing request, modifying the directedacyclic graph to enable the directed acyclic graph to satisfy theprocessing request.

In some embodiments, the directed acyclic graph is modified while datais being routed through the directed acyclic graph.

In some embodiments, modifying the directed acyclic graph includesinserting at least one additional instance of one of the data processingsub-units into the directed acyclic graph.

In some embodiments, modifying the directed acyclic graph includesinserting at least one new data processing sub-unit into the directedacyclic graph.

In some embodiments, modifying the directed acyclic graph includesmodifying the at least one of the operators positioned between theplurality of data processing sub-units.

In some embodiments, the method includes limiting a volume of datareceived at the data collection input from the at least one incomingdata stream data that is transmitted to the directed acyclic graph.

In some embodiments, the method includes: monitoring the throughput ofeach of the data processing sub-units; and adjusting the volume of datathat is transmitted to the directed acyclic graph based on the monitoredthroughput.

In some embodiments, the method includes: monitoring the performance ofeach of the data processing sub-units; identifying a performancedeficiency in the monitored performance of at least one of the dataprocessing sub-units; and modifying the directed acyclic graph inresponse to the identified performance deficiency.

In some embodiments, the at least one data output unit may include atleast one of a data analysis application and a real-time storageapplication.

In accordance with an aspect of this disclosure there is provided acomputer program product comprising a non-transitory computer-readablemedium having computer-executable instructions stored therein, thecomputer-executable instructions being executable by a processor toconfigure the processor to perform a method for processing data, wherethe method includes: receiving at least one incoming data stream;modifying the data in the incoming data stream using a plurality of dataprocessing sub-units, where each data processing sub-unit is operable toreceive an input data stream, modify data in the input data stream, andoutput an outgoing data stream that includes the modified data; routingthe data from the incoming data stream through a directed acyclic graphto at least one data output unit, where the directed acyclic graphincludes the plurality of data processing sub-units and a plurality ofoperators connecting the data collection input, the plurality of dataprocessing sub-units, and the at least one data output unit, and theoperators are operable to route the data through the directed acyclicgraph; where routing the data through the incoming data stream includes,for at least some first data processing units, transmitting data to eachdata processing sub-unit in the at least some first data processingunits using the User Datagram Protocol (UDP) and for at least somesecond data processing units transmitting data from each data processingsub-unit in the at least some second data processing units using UDP.

In another broad aspect, there is provided a system for processingnetwork flow monitoring data. The system includes a data collectioninput coupled to at least one incoming data stream of network monitoringmessages; a load balancing operator coupled to the data collectioninput, the load balancing operator operable to distribute the networkmonitoring messages amongst a plurality of downstream processing units;a plurality of collector processing sub-units coupled to the loadbalancing operator downstream from the load balancing operator, eachcollector processing sub-unit operable to generate network flow recordsfrom the received network monitoring messages, where the network flowrecords are generated in a first format; a plurality of parser sub-unitscoupled to the plurality of collector processing sub-units, each parsersub-unit operable to receive the network flow records from one of thecollector processing sub-units and to generate parsed network flowrecords by converting the received network flow records from the firstformat to a second format; and a combiner operator coupled to theplurality of parser sub-units, where the combiner operator is operableto combine the parsed network flow records from the plurality of parsersub-units into a synchronous output data stream.

In some embodiments, the load balancer operator is configured totransmit the network monitoring messages to the plurality of collectorprocessing sub-units using the User Datagram Protocol (UDP).

In some embodiments, each collector processing sub-unit is configured togenerate the network flow records in a JavaScript Object Notationformat.

In some embodiments, each collector processing sub-unit is configured totransmit the network flow records to the corresponding parser sub-unitusing the Transmission Control Protocol (TCP).

In some embodiments, each parser sub-unit is configured to generate theparsed network flow records in a CSV file format.

In some embodiments, each parser sub-unit is configured to transmit theparsed network flow records using the User Datagram Protocol (UDP).

In some embodiments, the system also includes at least one streamenrichment processing sub-unit coupled downstream from the combineroperator, each stream enrichment processing sub-unit operable togenerate enriched network flow records by inserting enrichment data intothe parsed network flow records.

In some embodiments, the at least one stream enrichment processingsub-unit includes a plurality of stream enrichment processing sub-unitsin parallel, and the system further includes: a switching operatorcoupling the plurality of stream enrichment processing sub-units to thecombiner operator, the switching operator operable to direct the parsednetwork flow records to a subset of the stream enrichment sub-units inthe plurality of stream enrichment sub-units.

In some embodiments, the plurality of stream enrichment processingsub-units includes a first subset of stream enrichment sub-units and asecond subset of stream enrichment sub-units, where the first subset ofstream enrichment sub-units corresponds to a first set of enrichmentdata and the second subset of stream enrichment sub-units corresponds toa different set of enrichment data; and the switching operator isoperable to selectively direct the parsed network flow records to thestream enrichment processing sub-units in one of the first subset andthe second subset while the stream enrichment processing sub-units areoperational.

In some embodiments, the system includes a stream output operatorcoupled to the at least one stream enrichment processing sub-unit, thestream output operator configured to output the enriched network flowrecords using the User Datagram Protocol (UDP).

In some embodiments, the system includes a duplicator operator coupleddownstream from the combiner operator, the duplicator operatorconfigured to duplicate the received network flow records; and aplurality of feature extraction processing sub-units coupled to theduplicator operator, each feature extraction processing sub-unitoperable to derive one or more network flow characteristics from theduplicated network flow records.

In some embodiments, the system includes a plurality of conditionaloperators coupled between the duplicator and the plurality of featureextraction processing sub-units, where each conditional operator isoperable to selectively direct the network flow records between theplurality of feature extraction processing sub-units by determining thatthe network flow record has a characteristics corresponding to theselected feature extraction processing sub-unit.

In some embodiments, the system includes a filter operator coupledupstream from the duplicator, where the filter operator is operable totransmit network flow records having a defined set of characteristics tothe duplicator and to prevent network flow records that do not have thedefined set of characteristics from being transmitted to the duplicator.

In some embodiments, the system includes at least one data output unitcoupled downstream of the combiner operator, where the at least one dataoutput unit includes at least one of a data analysis application and areal-time storage application.

In some embodiments, the system may include a compression sub-unitupstream from the real-time storage application.

In some embodiments, the real-time storage application may be configuredto store output data with time index data determined from the parsednetwork flow records.

In some embodiments the system may include a sequence of at least oneparser sub-unit, at least one enrichment sub-unit and at least onecompression, and the real-time storage application may be configured tostore output data from the sequence of the at least one parser sub-unit,at least one enrichment sub-unit and at least one compression with timeindex data for subsequent retrieval. In some embodiments, the time indexdata is determined by the at least one parser sub-unit.

In accordance with a broad aspect there is provided a method ofprocessing network flow monitoring data. The method includes: receivingat least one incoming data stream of network monitoring messages;distributing the network monitoring messages amongst a plurality ofdownstream processing units; generating, by a plurality of collectorprocessing sub-units in the plurality of downstream processing units,network flow records from the received network monitoring messages,wherein the network flow records are generated in a first format;transmitting the network flow records to a plurality of parserprocessing sub-units downstream from the collector processing sub-units;generating, by the plurality of parser processing sub-units, parsednetwork flow records by converting the received network flow recordsfrom the first format to a second format; and combining the parsednetwork flow records from the plurality of parser sub-units into asynchronous output data stream.

In some embodiments, the network monitoring messages are distributedamongst the plurality of collector processing sub-units using the UserDatagram Protocol (UDP).

In some embodiments, the network flow records are generated in aJavaScript Object Notation format.

In some embodiments, the network flow records are transmitted to theplurality of parser processing sub-units using the Transmission ControlProtocol (TCP).

In some embodiments, the parsed network flow records are generated in aCSV file format.

In some embodiments, the parsed network flow records are output from theparser processing sub-units using the User Datagram Protocol (UDP).

In some embodiments, the method includes generating enriched networkflow records by inserting enrichment data into the parsed network flowrecords.

In some embodiments, a plurality of stream enrichment processingsub-units are coupled in parallel to the synchronous output data streamand the method further includes: directing the parsed network flowrecords to a subset of the stream enrichment sub-units in the pluralityof stream enrichment sub-units.

In some embodiments, the plurality of stream enrichment processingsub-units includes a first subset of stream enrichment sub-units and asecond subset of stream enrichment sub-units, where the first subset ofstream enrichment sub-units corresponds to a first set of enrichmentdata and the second subset of stream enrichment sub-units corresponds toa different set of enrichment data, and the method further includesselectively directing the parsed network flow records to the streamenrichment processing sub-units in one of the first subset and thesecond subset while the stream enrichment processing sub-units areoperational.

In some embodiments, the method includes transmitting the enrichednetwork flow records using the User Datagram Protocol (UDP).

In some embodiments, the method includes duplicating the network flowrecords; transmitting the duplicated network flow records to a pluralityof feature extraction processing sub-units; and deriving, by eachfeature extraction processing sub-unit, one or more network flowcharacteristics from the duplicated network flow records.

In some embodiments, transmitting the duplicated network flow records tothe plurality of feature extraction processing sub-units includesselectively directing each network flow record between the plurality offeature extraction processing sub-units by determining that the networkflow record has a characteristic corresponding to the selected featureextraction processing sub-unit.

In some embodiments, the method includes filtering the network flowrecords prior to duplication, where network flow records having adefined set of characteristics are duplicated and network flow recordsthat do not have the defined set of characteristics are prevented frompassing through the filter.

In some embodiments, the method includes routing the output data streamto at least one data output unit, where the at least one data outputunit includes at least one of a data analysis application and areal-time storage application.

In accordance with a broad aspect there is provided a computer programproduct comprising a non-transitory computer-readable medium havingcomputer-executable instructions stored therein, the computer-executableinstructions being executable by a processor to configure the processorto perform a method of processing network flow monitoring data, wherethe method includes: receiving at least one incoming data stream ofnetwork monitoring messages; distributing the network monitoringmessages amongst a plurality of downstream processing units; generating,by a plurality of collector processing sub-units in the plurality ofdownstream processing units, network flow records from the receivednetwork monitoring messages, where the network flow records aregenerated in a first format; transmitting the network flow records to aplurality of parser processing sub-units downstream from the collectorprocessing sub-units; generating, by the plurality of parser processingsub-units, parsed network flow records by converting the receivednetwork flow records from the first format to a second format; andcombining the parsed network flow records from the plurality of parsersub-units into a synchronous output data stream.

It will be appreciated by a person skilled in the art that an apparatusor method disclosed herein may embody any one or more of the featurescontained herein and that the features may be used in any particularcombination or sub-combination.

These and other aspects and features of various embodiments will bedescribed in greater detail below.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the described embodiments and to show moreclearly how they may be carried into effect, reference will now be made,by way of example, to the accompanying drawings, in which:

FIG. 1 is a block diagram of a data processing computer network systemin accordance with an example embodiment;

FIG. 2A is a block diagram of a data stream processing system inaccordance with an example embodiment;

FIG. 2B is a block diagram of another data stream processing system inaccordance with an example embodiment;

FIG. 2C is a flowchart illustrating a method of processing streamingdata in accordance with an example embodiment;

FIGS. 3A-3D are block diagrams of example operators that may be used inthe data stream processing systems of FIGS. 2A and 2B in accordance withan embodiment;

FIGS. 4A-4E are block diagrams of further example operators that may beused in the data stream processing systems of FIGS. 2A and 2B inaccordance with an embodiment;

FIG. 5A is a block diagram of a control sub-system that may be used withthe data stream processing systems of FIGS. 2A and 2B in accordance withan embodiment;

FIG. 5B is a flowchart illustrating a method of updating a data streamprocessing system in accordance with an example embodiment;

FIG. 5C is a flowchart illustrating another method of updating a datastream processing system in accordance with an example embodiment;

FIGS. 6A-6B are block diagrams of data stream processing segments thatmay be used in the data stream processing systems of FIGS. 2A and 2B inaccordance with an embodiment;

FIG. 6C is a flowchart illustrating a method of processing network datain accordance with an example embodiment;

FIGS. 7A-7C are block diagrams of data processing segments that may beused in the data stream processing systems of FIGS. 2A and 2B inaccordance with an embodiment;

FIG. 8 is a block diagram of another data processing segment that may beused in the data stream processing systems of FIGS. 2A and 2B inaccordance with an embodiment;

FIG. 9 is a block diagram of a data analysis segment that may be used inthe data processing computer network system of FIG. 1 in accordance withan embodiment; and

FIG. 10 is a block diagram of another data analysis segment that may beused in the data processing computer network system of FIG. 1 inaccordance with an embodiment.

The drawings, described below, are provided for purposes ofillustration, and not of limitation, of the aspects and features ofvarious examples of embodiments described herein. For simplicity andclarity of illustration, elements shown in the drawings have notnecessarily been drawn to scale. The dimensions of some of the elementsmay be exaggerated relative to other elements for clarity. It will beappreciated that for simplicity and clarity of illustration, whereconsidered appropriate, reference numerals may be repeated among thedrawings to indicate corresponding or analogous elements or steps.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Various systems or methods will be described below to provide an exampleof an embodiment of the claimed subject matter. No embodiment describedbelow limits any claimed subject matter and any claimed subject mattermay cover methods or systems that differ from those described below. Theclaimed subject matter is not limited to systems, methods or computerprogram products having all of the features of any one system, method orcomputer program product described below or to features common tomultiple or all of the systems or methods described below. It ispossible that a system or method described below is not an embodimentthat is recited in any claimed subject matter. Any subject matterdisclosed in a system or method described below that is not claimed inthis document may be the subject matter of another protectiveinstrument, for example, a continuing patent application, and theapplicants, inventors or owners do not intend to abandon, disclaim ordedicate to the public any such subject matter by its disclosure in thisdocument.

Furthermore, it will be appreciated that for simplicity and clarity ofillustration, where considered appropriate, reference numerals may berepeated among the figures to indicate corresponding or analogouselements. In addition, numerous specific details are set forth in orderto provide a thorough understanding of the embodiments described herein.However, it will be understood by those of ordinary skill in the artthat the embodiments described herein may be practiced without thesespecific details. In other instances, well-known methods, procedures andcomponents have not been described in detail so as not to obscure theembodiments described herein. In addition, the description is not to beconsidered as limiting the scope of the embodiments described herein.

The terms “an embodiment,” “embodiment,” “embodiments,” “theembodiment,” “the embodiments,” “one or more embodiments,” “someembodiments,” and “one embodiment” mean “one or more (but not all)embodiments of the present invention(s),” unless expressly specifiedotherwise.

The terms “including,” “comprising” and variations thereof mean“including but not limited to,” unless expressly specified otherwise. Alisting of items does not imply that any or all of the items aremutually exclusive, unless expressly specified otherwise. The terms “a,”“an” and “the” mean “one or more,” unless expressly specified otherwise.

It should also be noted that the terms “coupled” or “coupling” as usedherein can have several different meanings depending in the context inwhich these terms are used. For example, the terms coupled or couplingmay be used to indicate that an element or device can electrically,optically, or wirelessly send data to another element or device as wellas receive data from another element or device.

It should be noted that terms of degree such as “substantially”, “about”and “approximately” as used herein mean a reasonable amount of deviationof the modified term such that the end result is not significantlychanged. These terms of degree may also be construed as including adeviation of the modified term if this deviation would not negate themeaning of the term it modifies.

Furthermore, any recitation of numerical ranges by endpoints hereinincludes all numbers and fractions subsumed within that range (e.g. 1 to5 includes 1, 1.5, 2, 2.75, 3, 3.90, 4, and 5). It is also to beunderstood that all numbers and fractions thereof are presumed to bemodified by the term “about” which means a variation of up to a certainamount of the number to which reference is being made if the end resultis not significantly changed.

The example embodiments of the systems and methods described herein maybe implemented as a combination of hardware or software. In some cases,the example embodiments described herein may be implemented, at least inpart, by using one or more computer programs, executing on one or moreprogrammable devices comprising at least one processing element, and adata storage element (including volatile memory, non-volatile memory,storage elements, or any combination thereof). These devices may alsohave at least one input device (e.g. a pushbutton keyboard, mouse, atouchscreen, and the like), and at least one output device (e.g. adisplay screen, a printer, a wireless radio, and the like) depending onthe nature of the device.

It should also be noted that there may be some elements that are used toimplement at least part of one of the embodiments described herein thatmay be implemented via software that is written in a high-level computerprogramming language such as object oriented programming. Accordingly,the program code may be written in C, C++ or any other suitableprogramming language and may comprise modules or classes, as is known tothose skilled in object oriented programming. Alternatively, or inaddition thereto, some of these elements implemented via software may bewritten in assembly language, machine language or firmware as needed. Ineither case, the language may be a compiled or interpreted language.

At least some of these software programs may be stored on a storagemedia (e.g. a computer readable medium such as, but not limited to, ROM,magnetic disk, optical disc) or a device that is readable by a generalor special purpose programmable device. The software program code, whenread by the programmable device, configures the programmable device tooperate in a new, specific and predefined manner in order to perform atleast one of the methods described herein.

Furthermore, at least some of the programs associated with the systemsand methods of the embodiments described herein may be capable of beingdistributed in a computer program product comprising a computer readablemedium that bears computer usable instructions for one or moreprocessors. The medium may be provided in various forms, includingnon-transitory forms such as, but not limited to, one or more diskettes,compact disks, tapes, chips, and magnetic and electronic storage. Thecomputer program product may also be capable of distribution through acommunication network such as the Internet.

In order to protect communication networks against issues such as faultsor security breaches, analysis and monitoring applications can be usedto assess the status of the communication network. Data about thenetwork, such as network flow data, syslog event data, DNS request data,and network topology data from various data sources can be collected onan ongoing basis as a stream of network related data. The networkrelated data can be analyzed and monitored in an effort to detectnetwork characteristics including events of interest such as faults,anomalies and intrusions. The data streams can also be monitored toidentify and learn about other network characteristics such as patternsof behavior, demand, usage, and attacks among other information that maybe useful in ensuring that the communication network remains secure andoperational. The collected data may also be stored for later access andplayback, e.g. for auditing purposes and/or to identify factors that mayhave contributed to a system fault.

As individuals and organizations continue to become increasingly relianton communication networks to control and manage their day-to-day livesand business operations, the volume of data relevant to networkmonitoring and analysis will continue to increase. To ensure thatnetwork operations can be monitored effectively, it is important thatlarge volumes of data can be assimilated and processed to providenetwork analysis and monitoring applications with meaningful input data.Real-time monitoring of network operations and security are also crucialin preventing or mitigating damage that may result from network faultsor intrusions. Accordingly, rapid processing of network monitoring datamay ensure that network analysis and monitoring application can providereal-time, or near real-time, feedback on the status of thecommunication network (and potential threats thereto).

Numerous different potential security and operational defects may arisein a communication network. Accordingly, numerous different analysisapplications may be used to monitor the network status. The variousanalysis applications may require different types of input data (e.g.data in different formats or with different informational content) inorder to detect a particular type of network event. Accordingly,incoming data streams may be processed to provide analysis applicationswith the correct type of data.

Embodiments described herein may provide systems, methods and computerprogram products that can be used to collect data from one or more datastreams and extract information relevant to downstream monitoring andanalysis applications. Embodiments described herein may collect andprocess one or more streams of input data and output one or more streamsof processed data usable by the monitoring and/or analysis applications.The streams of processed data may also be stored for subsequent analysisby monitoring and/or analysis application. For example, the processeddata may be stored with time index data to facilitate subsequentanalysis of the processed data. The systems and methods described hereinmay perform various operations, such as selecting data streams and/ordata items, extracting informational content from data streams and/ordata items, and enriching data streams and/or data items for example.

Embodiments described herein may be configured to provide a data streamprocessing system. The data stream processing system can include one ormore data inputs, each coupled to a source of streaming data. The datastream processing system can also include one or more data outputs. Eachdata output can be coupled to a data output unit. The data output unitcan include at least one of a data analysis and/or monitoringapplication and a data storage application.

The data stream processing system can also include a plurality of dataprocessing sub-units. Each data processing sub-unit may be operable toreceive an input data stream and generate an outgoing data stream bymodifying data in the input data stream. The data processing sub-unitcan modify the data in the input stream so that the outgoing data streamcan have different informational content or be formatted differently ascompared to the input data stream.

The data processing system also includes a plurality of operatorsconnecting the data inputs, the data processing sub-units and the dataoutputs. The plurality of data processing sub-units can be connected bythe plurality of operators in a directed acyclic graph (i.e. aunidirectional processing graph). The plurality of data processingsub-units, and their interconnection in the processing graph, can bedefined to process the incoming data streams and to generate processeddata required by each of the downstream monitoring and analysisapplications.

The plurality of data processing sub-units may be loosely coupled by theplurality of operators. That is, the plurality of operators may beadjustable to re-direct the flow of data between the plurality of dataprocessing sub-units. The operators may be reconfigurable in real-time,i.e. while the system is operational and without interrupting operationof the data stream processing system.

The system may also include a central control system. The centralcontrol system can be configured to control and manage the deployment ofthe plurality of data processing sub-units. The control system may alsobe configured to control the deployment of the plurality of operators,and the interconnections between the data processing sub-units and theplurality of operators.

The control system may include one or more system monitoring components.The system monitoring components can generate monitoring data indicatingthe status of a data processing sub-unit or operator. For instance, insome cases each data processing sub-unit and/or each operator may havean associated monitoring component. Alternatively, each monitoringcomponent may monitor a plurality of data processing sub-units and/oroperators. The monitoring components may provide the monitoring data tothe control system to enable the control system to re-configure theprocessing system as required.

The system monitoring components can monitor the status and performanceof the data processing sub-units and/or operators. For instance, themonitoring components may monitor various performance characteristicssuch as input date rate, output data rate, buffer fill etc. The controlsystem may then identify various system performance conditions, such asload imbalances, input data overloads, low output data rates, sub-unitfaults etc. that may indicate that the processing system should bere-configured. The control system may then update the deployment of thedata processing sub-units to account for the identified performancedefects.

The control system may update the processing system by scaling the dataprocessing sub-units to account for the identified performance issues.For instance, additional instances of data processing sub-units may bedeployed to increase the resources available for a given processingoperation. The control system may also update the interconnection of theoperators to ensure that data is routed to the newly deployed dataprocessing sub-unit instance(s).

In some embodiments, the control system may be configured toautomatically update the configuration of the data processing system inresponse to monitored system conditions and/or application requirements.Alternatively, modifications to the data stream processing system may beinitiated by an administrator in response to prompts from the controlsystem.

In some embodiments, some of the operators and/or data processingsub-units may be configured to transmit data using the User DatagramProtocol (UDP). This may ensure that data is transmitted between dataprocessing sub-units rapidly. UDP may also allow the processing systemto be updated on the fly, without requiring additional overhead inestablishing communication links. This may allow operators to redirectdata streams to new and/or different data processing sub-units while thesystem remains operational.

Referring now to FIG. 1, there is provided is a block diagram of an dataprocessing computer network system 100 in accordance with an exampleembodiment. System 100 is an example of a data processing system thatmay be used to monitor and analyze a communication network.

Computer network system 100 generally includes a plurality of computersconnected via one or more data communication networks, which may beconnected to the Internet. In general, however, the computer networksystem includes a data stream processing system 105, at least one datasource 110, and at least one data output application 120 connected viathe network. Optionally, one or more of the data output applications 120may be implemented as a data evaluation application, such as a dataanalysis application or data monitoring application. Alternately or inaddition, one or more of the data output applications 120 may beimplemented as a data storage application.

In some cases, the connection between network and the Internet may bemade via a firewall server (not shown). In some cases, there may bemultiple links or firewalls, or both, between the network and theInternet. System 100 may also be implemented using multiple networks orvirtual networks, which can be internetworked or isolated. These havebeen omitted for ease of illustration, however it will be understoodthat the teachings herein can be applied to such systems. The networksmay be constructed from one or more computer network technologies, suchas IEEE 802.3 (Ethernet), IEEE 802.11 and similar technologies.

As shown in FIG. 1, system 100 includes a plurality of data sources 110a-110 n. Each data source 110 a-110 n can be configured to provide acorresponding stream 111 a-111 n of data. The data sources 110 a-110 nmay include various different types of data, such as network flow data(NetFlow, IPFix, sFlow, Bro Flows, etc.), syslog event data, DNS requestlogs, host event logs, collected threat information, and data fromnetwork probes, Internet of Things applications, and other datagenerating devices.

The system 100 also includes a data stream processing system 105. Thedata stream processing system 105 can receive the plurality of datastreams 111 from the plurality of data sources 110. The data streamprocessing system 105 can be configured to process the received datastreams to generate output data streams 121 a-121 m. Each output datastream 121 a-121 m can be provided to an analysis/monitoring applicationcoupled to the data stream processing system 105, such as applications120 a-120 m. Alternately or in addition, an output data stream 121 a-121m can be provided to a data storage application coupled to the datastream processing system 105.

The applications 120 can include one or more applications configured todetect network characteristics such as events of interest in thereceived data. Examples of events of interest can include faults,anomalies, and intrusions/attacks in the network being monitored. Theapplications 120 may also be configured to identify and learn aboutother network characteristics such as patterns of demand, usage, andattacks on the network(s) being monitored.

The applications 120 can also include data storage applications. Thedata storage applications can be configured to perform real-time storageof the some, all, or selected portions of the data streams 111. In somecases, an application 120 may integrate both data analysis and datastorage functionality.

The applications 120 may provide network monitoring feedback on theidentified network characteristics, such as identifying an event ofinterest (e.g. faults, anomalies, and intrusions) and/or identifyingpatterns of behavior, demand, usage, and attacks. The network monitoringfeedback can include a prompt or prompts for administrator intervention.

In order to detect events of interest, the applications 120 may requirethe data streams 111 to be pre-processed. The data stream processingsystem 105 can be configured to process/manipulate the received datastreams 111 to generate the processed data required by the applications120. Examples of processing operations performed by the data streamprocessing system 105 can include data streams being parsed, filtered,matched, enriched, transformed (e.g. compressed and/or expanded), andextracting features for example. The processed data streams 121 can thenbe analyzed by the applications 120. Alternately or in addition, theprocessed data streams 121 may be stored by the applications 120 (e.g.in non-volatile storage memory). The stored processed data streams maysubsequently be accessed/retrieved for evaluation (e.g. fault analysis,auditing etc.).

Each incoming data stream 111 can arrive at a data collection input, oringress interface, of the data stream processing system 105. Eachincoming data stream may include blocks of data encapsulated in avariety of protocols, such as UDP, TCP, SCTP, Protocol Buffers, ApacheThrift, and AMQP for example. The data stream processing system 105 caninclude a plurality of data processing sub-units that perform variousfunctions, such as parsing, filtering, matching, enriching, transforming(e.g. compressing, expanding etc.), anonymizing, etc. in order toextract data required by the applications 120. The data streamprocessing system 105 can then output a stream 121 of extracted data toeach application 120.

The incoming data streams 111 coupled to the data stream processingsystem 105 may contain the information required for a variety ofdifferent analysis applications. However, the specific format and/orcontent of the data required by the individual applications 120 mayvary. Accordingly, the data stream processing system 105 can define aplurality of processing sequences (or sub-graphs) for the applications120. The data stream processing system 105 may be defined to performsimultaneous extraction/processing of data required by a plurality ofapplications 120.

In some cases, the data stream processing system 105 may defineseparate, or at least partially separate, data processingsequences/sub-graphs for each application 120. Alternatively, there maybe cases where two or more applications 120 require the same data, or atleast some of the same data. Accordingly, the data stream processingsystem 105 may define at least partially overlapping processingsequences for the applications 120 requiring overlapping data.Optionally, the data analysis applications may each includecorresponding data storage applications to ensure that the data requiredfor each analysis application is stored for later access and retrieval.The corresponding data storage applications may be connected to the sameoutput data stream and/or a duplicated output data stream as thecorresponding data analysis application. In some cases, only a subset ofdata analysis applications or data output streams may be coupled to adata storage application.

In some embodiments, the data stream processing system 105 may bedynamically reconfigurable. That is, the data stream processing system105 may be configured to adapt to changes in the incoming data streams111 and/or changes in application requirements. The data streamprocessing system 105 may be configured to dynamically update theprocessing graph defined therein to account for the changes in networkoperations and/or monitoring requirements.

The components of system 100 may be implemented using a combination ofhardware and software. In system 100, computers and computing devicesmay be connected to a network or a portion thereof via suitable networkinterfaces. The components of system 100, such as the data sources 110,data stream processing system 105 and applications 120 may beimplemented using computers, such as laptop, desktop, and/or servercomputers. These computers may connect to a network and one another viaa wired Ethernet connection or a wireless connection. The components ofsystem 100 may also connect via the Internet.

In general, the components of system 100 may be implemented using one ormore computers, each having a processor, volatile memory andnon-volatile storage memory, at least one network interface. In somecases, the computers may also include input devices such as a keyboardand trackpad, output devices such as a display and speakers, and variousother input/output devices where those devices may be operated by a useror administrator.

Data stream processing system 105 can be provided by one or morecomputer or computer servers. Data stream processing system 105 includesat least one processor, volatile and non-volatile memory, at least onenetwork interface, and may have various other input/output devices. Asshown, data stream processing system 105 is linked to data sources 110using a network. In some cases, the data stream processing system 105may be linked to the data sources 110 using a local network or closednetwork. However, in other embodiments, the data sources 110 and/or datastream processing system 105 may be linked via the Internet.

In some embodiments, the data stream processing system 105 may beprovided using virtual machines and/or containers corresponding theresources required for a given implementation. Example configurations ofthe data stream processing system 105 are described in greater detailwith reference to FIGS. 2A and 2B below.

As used herein, the term “software application” or “application” refersto computer-executable instructions, particularly computer-executableinstructions stored in a non-transitory medium, such as a non-volatilememory, and executed by one or more computer processors. The computerprocessor(s), when executing the instructions, may receive inputs andtransmit outputs to any of a variety of input or output devices to whichit is coupled.

The software application may be associated with an applicationidentifier that uniquely identifies that software application. In somecases, the application identifier may also identify the version andbuild of the software application. Within an organization, a softwareapplication may be recognized by a name by both the people who use it,and those that supply or maintain it.

A software application can be, for example, a monolithic softwareapplication, built in-house by the organization and possibly running oncustom hardware; a set of interconnected modular subsystems running onsimilar or diverse hardware; a software-as-a-service applicationoperated remotely by a third party; third party software running onoutsourced infrastructure, etc. In some cases, data stream processingsystem 105 may be provided as a software application that can beintegrated into a computer network system such as system 100.Alternatively, the data stream processing system 105 may be implementedas a remote or cloud-based processing system that is in communicationwith a network to be monitored.

It will be understood by those of skill in the art that referencesherein to data stream processing system 105 (or components thereof) ascarrying out a function or acting in a particular way imply that aprocessor or processors is/are executing instructions (e.g., a softwareprogram) stored in memory and possibly transmitting or receiving inputsand outputs via one or more interface. The memory may also store datainput to, or output from, processor in the course of executing thecomputer-executable instructions.

In some embodiments, the data stream processing system 105 may alsoinclude, or be linked to, an administrator computer. For example, thedata stream processing system 105 may be directly linked to anadministrator computer, for example, via a Universal Serial Bus,Bluetooth™ or Ethernet connection. Alternatively, data stream processingsystem 105 may be linked to the administrator computer via a localnetwork or, in some cases, the Internet. In some other cases, theadministrator computer and data stream processing system 105 may beintegrated or co-located. The administrator computer may allow anadministrator of the system 100 to monitor the system 100 and tomanually update the data sources 110, applications 120 and configurationof the data stream processing system 105. In some cases, theadministrator computer may be omitted, for instance where the datastream processing system 105 is configured to operate autonomously.

The administrator computer can include a processor, a display, a memory,a communication interface and a database. The administrator computer maybe provided as a desktop computer, laptop computer, or mobile devicesuch as a smartphone or tablet that is operable to communicate with thedata stream processing system.

The processor of the administrator computer is a computer processor,such as a general purpose microprocessor. In some other cases, processormay be a field programmable gate array, application specific integratedcircuit, microcontroller, or other suitable computer processor.

The processor is coupled to the display, which is a suitable display foroutputting information and data as needed by various computer programs.In particular, display may display a graphical user interface (GUI). Theadministrator computer may execute an operating system, such asMicrosoft Windows™ GNU/Linux, or other suitable operating system.

The processor is coupled, via a computer data bus, to the memory. Memorymay include both volatile and non-volatile memory. Non-volatile memorystores computer programs consisting of computer-executable instructions,which may be loaded into volatile memory for execution by processor asneeded.

The memory on the administrator computer may store a softwareapplication referred to herein as a system-monitoring dashboard. Thesystem-monitoring dashboard may be configured to monitor the operationof data stream processing system 105, and to enable an administrator tomake modifications to the data stream processing system 105.

The system 100 may also communicate detected events of interest oridentified patterns to the administrator computer. The system-monitoringdashboard may provide graphical user interfaces to allow anadministrator to review the identified events and/or patterns. Thesystem-monitoring dashboard may allow the administrator to implementmitigating actions in response to the feedback from the system 100.

In some cases, automated network protection responses may be performeddirectly by data processing system 100 without requiring communicationto the administrator computer.

Referring now to FIG. 2A, there is shown a block diagram of a datastream processing system 200 a in accordance with an example embodiment.In some embodiments, data stream processing system 200 a may be used toprovide the data stream processing system 105 shown in FIG. 1.

Data stream processing system 200 a is an example of a processing systemin which incoming data streams 111 are connected to processed datastream outputs 121 through a plurality of processing components that areinterconnected in a feed-forward graph structure. FIG. 2A illustrates asimplified example that includes only two data processing sub-units 220a and 220 b.

As shown in FIG. 1, the data stream processing system 200 a can includea data collection input 205. The data collection input 205 can beconnected to at least one incoming data stream 111. As shown in FIG. 2a, the data collection input 205 is connected to a plurality of incomingdata streams 111 a-111 n. As explained above, the incoming data streams111 may include various types of data, such as network flow data(NetFlow, IPFix, sFlow, Bro Flows, etc.), syslog event data, DNS requestlogs, host event logs, collected threat information, and data fromnetwork probes, Internet of Things applications, and other datagenerating devices. The incoming data streams 111 a-111 n may providedata using various different communication protocols.

The data stream processing system 200 a can also include a plurality ofdata processing sub-units 220. In FIG. 2A, only two data processingsub-units 220 a and 220 b are shown, although it should be apparent thatmany more data processing sub-units 220 may be included in embodimentsof data stream processing system 200 a.

Each data processing sub-unit 220 can be preceded by, and followed by, adata operator 210 (i.e. each data processing sub-unit is connected to atleast one upstream data operator and at least one downstream dataoperator). In some cases, each data processing sub-unit 220 may bedirectly connected to one or both of the upstream and downstreamoperator. However, in some cases a sequence of two or more dataprocessing sub-units 220 may be directly connected without requiring anintervening operator 210 (see e.g. FIG. 6A).

Each data processing sub-unit 220 has a data input 222 and a data output224. The data processing sub-unit 220 receives an input data stream atits data input 222. The data processing sub-unit 220 can then produce anoutput data stream by manipulating and modifying the data in the inputdata stream. The output data stream, including the modified data, canthen be output from the data output 224.

Data processing sub-units 220 may be configured to modify data from theinput data stream in different ways. In some cases, a data processingsub-unit 220 may be configured to generate the output data stream as adirect modification of the input data stream, e.g. by parsing, matching,filtering, subsampling, enriching, anonymizing, reformatting, matching,transforming (e.g. compressing, expanding) etc. the input data stream.These modifications may include inserting additional data into the inputdata stream. In some cases, a data processing sub-unit 220 may beconfigured to generate the output data stream by modifying the inputdata stream to extract additional information therefrom, for instance bytransforming the input data stream via feature extraction,classification, or detection techniques for example.

The data processing sub-units 220 may be implemented as software,hardware or a combination of both. For instance, the data processingsub-units 220 may be implemented as software components running oncontainers, virtual machines, or directly on hardware (i.e. bare metal).

In some embodiments, the data processing sub-units 220 may beimplemented using high performance languages such as C/C++. This mayprovide system 200 a with reduced delays and allow for improvedperformance. In other embodiments, the data processing sub-units 220 maybe implemented using programmable or special purpose hardware. Theimplementation of the data processing sub-units 220 may depend on therequirements of a given system, such as throughput and performancerequirements.

The data stream processing system 200 a also includes a plurality ofoperators 210. The plurality of operators 210 can connect the datacollection input 205, the plurality of data processing sub-units 220 andone or more data output units, such as data analysis applications and/ordata storage applications (not shown in FIG. 2A). The operators 210 candirect the flow of data into and out of each data processing sub-unit220. In directing the flow of data, the operators 210 can leave intactthe informational content of the data that is being passed through. Theoperators 210 can also provide the required connectivity between dataprocessing sub-units, for instance, by modifying the communicationprotocol used to transmit data.

The plurality of operators 210 and data processing sub-units 220 can bearranged into a directed acyclic graph in which data flows from the datacollection input 205 through the plurality of data processing sub-units220 and to the data output units (e.g. data analysis applications and/ordata storage applications) via the plurality of operators 210. Data canbe provided to the data output units (e.g. data analysis applicationsand/or data storage applications) as a processed output data stream.

Various different types of operators may be used in system 200 a. Ingeneral, the operators may be grouped into two types of operators,compositional operators and connectivity operators.

Compositional operators may be used to direct, and re-direct, the flowof data between different data processing sub-units 220. Some examplesof compositional operators are shown in FIGS. 3A-3D described hereinbelow. In some embodiments, variants of the compositional operators maybe provided that operate using different communication formats and/orprotocols.

Connectivity operators may be used to provide required communicationconnectivity between subsequent data processing sub-units 220. That is,the connectivity operators may provide translation operations thatmodify the format and/or protocol that is used to transmit the databetween subsequent data processing sub-units 220. Some examples ofconnectivity operators are shown in FIGS. 4A-4E described herein below.

In some cases, operators may be configured to provide both compositionaland connectivity functions.

Data stream processing system 200 a provides a simplified example of adirected acyclic graph that includes a plurality of operators 210 anddata processing sub-units 220. As shown in FIG. 2A, data flows from thedata input 205 and into data processing sub-units 220 a via operator 210a. The output data stream from data processing sub-unit 220 a then flowsto data processing sub-unit 220 b via operator 210 b.

In general, the plurality of incoming data streams 111 can be connectedto one or more outgoing processed data streams 121 through a pluralityof data processing sub-units 220 using operators 210. FIG. 2Aillustrates a simplified example with a direct sequence of twoprocessing sub-units 220 a and 220 b. In general, however, there may bevarious different branching processing subgraphs or completelyindependent processing sequences/subgraphs downstream from the datainput 205. The totality of the processing sequences/subgraphs can bearranged as a directed acyclic graph (i.e. a feedforward graph).

The specific processing subgraphs/processing paths from the input 205 toa particular output stream 121 can be defined to include the pluralityof data processing sub-units 220 that operate in conjunction to extractthe information required for the application connected to that outputstream 121. In some cases, the processing subgraphs for two or moreapplications may at least partially overlap, for instance where at leastsome of the same processing steps are required for those applications(see e.g. FIG. 7C).

The data stream processing system 200 a can be configured so that atleast some of the data processing sub-units 220 (and, accordingly, someof the operators 210) can transmit data using the User Datagram Protocol(UDP). The data outputs of these data processing sub-units 220 can bedirectly connected to a sub-unit operator that receives data using UDP.Similarly, at least some of the data processing sub-units 220 mayreceive data using UDP. The data inputs of these data processingsub-units 220 can be directly connected to a sub-unit input operatorthat transmits data using UDP.

In some cases, for at least some of the data processing sub-units 220the data input 222 of each of those data processing sub-units 220 can bedirectly connected to an upstream operator 210 that transmits data usingthe User Datagram Protocol (UDP) and the data output of each of thosedata processing sub-units 220 can also be directly connected to adownstream operator 210 that transmits data using UDP.

Configuring components of the data stream processing system 200 a totransmit data using UDP may provide the system 200 a with low-latencycommunications. The operators 210 and data processing sub-units 220communicating using UDP may avoid bottlenecks that might otherwise becaused by the use of communication protocols that require initialhandshake routines or messaging overhead to transmit messages.

The operators 210 can provide loose couplings between the plurality ofdata processing sub-units 220 in system 200 a. This may allow the dataprocessing sequences and subgraphs to be updated while the system isoperational. The plurality of operators 210 may be dynamicallyreconfigurable to adapt to changes in the operation and/or requirementsof system 200 a. That is, the operators 210 may change the downstreamdata processing sub-unit 220 to which they direct data while data isflowing therethrough.

In embodiments of system 200 a, the data processing sub-units 220 and/oroperators 210 may be scalable. That is, additional instances of dataprocessing sub-units 220 may be added (or removed) from the system 200a. Instances of data processing sub-units 220 may be added/removed toaccount for changes in the system 200 a, such as increases or decreasesin the data rate from one or more incoming data streams 111 and/orchanges to the applications being served.

Redirecting, or adjusting the direction of data flow, while the system200 a is operational may facilitate real-time scaling of system 200 a.In response to determining that an additional instance of a dataprocessing sub-unit is required, a new data processing sub-unit 220 canbe deployed. The data processing sub-unit can be coupled to a downstreamoperator that is configured to receive data from that data processingsub-unit. Once the data processing sub-unit 220 is deployed, theupstream operator 210 can re-direct some or all of the data flowingtherethrough to the newly deployed data processing sub-unit. In somecases, the flow of data may be re-directed to balance the data loadbetween different data processing sub-units, including any newlydeployed data processing sub-units 220.

In some cases, redirecting the flow of data may include changing thedirection of data flow from one data processing sub-unit 220 to anotherdata processing sub-unit. This may occur, for instance, where the dataprocessing sub-unit is being replaced with an instance of an updateddata processing sub-unit or a different data processing sub-unit.Following deployment of the new data processing sub-unit, the upstreamoperator 210 can dynamically stop transmitting data to the old instanceand begin transmitting to the newly deployed instance.

The operators 210 may be configured to transmit data using variouscommunication protocols. For instance, some of the operators 210 can beconfigured to transmit data using UDP. This may facilitate dynamicre-direction of the data flow, as UDP does not require (i.e. may omit)any handshaking operations prior to initiating communication. The UDPoutput stream of an operator 210 can be redirected to a new or differentdata processing sub-unit 220 without interrupting data transmission orrequiring an initial handshaking protocol—in other words, the UDP outputstream of an operator 210 can be redirected to a new or different dataprocessing sub-unit 220 while data transmission is ongoing and/oromitting an initial handshaking protocol.

In some embodiments, one or more operators 210 may use TCP tocommunicate with downstream operators 210 and/or processing sub-units220. This may ensure reliable data transfer while avoiding/minimizingdeletions, errors and insertions. Operators 210 that use TCP can beconfigured to transmit streams of data bytes, although the boundariesbetween block of data bytes (e.g. data files) may not be identified inthe TCP stream. In some cases, the use of TCP may be limited to datatransmission between remote servers.

In some embodiments, one or more operators 210 may use pipe-basedcommunication protocols. Pipes define a method for interprocesscommunication that is arranged in a First-In-First-Out (FIFO)configuration. An operator 210 can write output data to a first, write,end of the pipe and the data at the first end of the pipe can bebuffered until it is read by the downstream operator 210 or processingsub-unit 220 that is coupled to the second, read, end of the pipe.

As shown in FIG. 2A, each of the operators 210 and data processingsub-units 220 can be connected to a corresponding monitoring agent211/221. In system 200 a, there is a one-to-one relationship between themonitoring agents 211/221 and the operators 210 and data processingsub-units 220. Alternatively, a monitoring agent may be configured tomonitor multiple operators 210 and/or data processing sub-units 220. Forinstance, an individual monitoring agent may be configured to monitor aset of parallel data processing sub-unit instances that each correspondto the same data processing sub-unit operation.

The monitoring agents 211/221 can monitor the performance of theoperator 210 or data processing sub-unit 220 to which it is connected.For instance, the monitoring agent may monitor the resource utilizationand/or performance of a given data processing sub-unit, such as CPUutilization, memory utilization, service rate, buffer fill andpacket/message loss for example. The performance data gathered by themonitoring agents 211/221 can be used to determine whether, and how, thecomponents or connections of system 200 a should be updated. Forinstance, where a monitoring agent 221 determines that the performanceof one of the data processing sub-units 220 is poor (e.g. there is veryhigh resource utilization, very high buffer fill, high pack loss, lowthroughput etc.), it may be necessary to replace that data processingsub-unit 220 and/or add an additional instance of that data processingsub-unit to account for the performance defect. For instance, themonitoring agents 211/221 may generate alarm signals when performancethresholds are met or exceeded.

The data processing system 200 a can also include a central controlcomponent. An example of a central control component is described infurther detail below with reference to FIG. 5A. The central controlcomponent can be connected to the monitoring agents 211/221. Eachmonitoring agent 211/221 can transmit monitoring data to the centralcontroller.

The monitoring agents 211/221 may transmit monitoring data to thecentral controller on an ongoing or continual basis. This may providethe central controller with an ongoing overview of system operations. Insome cases, the monitoring agents 211/221 may transmit monitoring datato the central controller in response to queries from the centralcontroller. For instance, the controller may poll the monitoring agents211/221 on a periodic basis to provide a periodic status update on theoperations of system 200 a. The monitoring agents 211/221 can alsotransmit alarm messages to the controller.

The controller may update system 200 a in response to the monitoringdata from the agents 211/221. For instance, the controller mayadd/remove data processing sub-units 220 and/or update operators 210 toadjust the connections between data processing sub-units 220.

In some embodiments, the controller may update the system 200 aautomatically to account for deviations from the required performance.Alternatively, the controller may generate prompts for an administratoruser to update the system 200 a. In some cases, the controller mayidentify potential updates to be approved or denied by an administratoruser.

In some embodiments, some or all of the data processing sub-units 220may include a data buffer at the data input 222. The size of the databuffer may be configured based on the data rate expected for system 200a. For implementations in which the data rate is expected to berelatively constant or steady, the data buffer size may be reduced. Therequired buffer size may depend on the throughput as well as the“burstiness” of data arriving at a given data processing sub-unit.Higher throughput and burstiness may require larger buffer size toaccommodate surges in arrivals.

The controller may monitor the buffer usage of each of those databuffers (e.g. using monitoring agents 221). Based on the monitoredbuffer usage the controller may modify the acyclic graph defined bysystem 200 a to improve system operations and/or resource utilization.

In some embodiments, a variable buffer size may be used for some dataprocessing sub-units 220. The buffer size may be adjustable by thecontroller in response to determining the buffer usage is increasing ata threshold rate and/or has reached a fill threshold. Additionally oralternatively, the controller may monitor the buffer usage to determinethat the processing graph should be modified. In some such instances,the buffer sizes may be static.

In some cases, one or more usage thresholds may be defined for thebuffers of the data processing sub-units 220. The controller may updatethe acyclic graph defined by system 200 a in response to determiningthat the buffer for one of the data processing sub-units 220 has reachedthat usage threshold.

For example, in some cases a usage threshold may be defined as a certainpercentage of buffer fill (e.g. 80% or 90%). When the monitored bufferusage for a given data processing sub-unit reaches the usage threshold,the controller may determine that an additional instance of that dataprocessing sub-unit is required. The controller may then modify theacyclic graph by adding an additional instance of that data processingsub-unit. The controller may then update the upstream operator(s) toredirect at least some data through the newly added data processingsub-unit instance.

In other cases, a usage threshold may be defined based on a change inbuffer usage for a given data processing sub-unit. For instance, wherethe controller determines that the buffer usage of a given dataprocessing sub-unit has increased by a defined rate, the controller mayagain determine that an additional instance of that data processingsub-unit is required and update the processing graph accordingly.

Referring now to FIG. 2B, shown therein is a block diagram showinganother example embodiment of a data stream processing system 200 b. Insome embodiments, data stream processing system 200 b may be used toprovide the data stream processing system 105 shown in FIG. 1.

Data stream processing system 200 b is an example of a processing systemin which the processing graph is defined as a plurality of dataprocessing segments 240 a-240 e (also referred to as a processing blockor service block). The deployment and operation of individual elementswithin data processing system 200 a may be generally implemented in amanner analogous to system 200 a. In system 200 b, each of the dataprocessing segments 240 includes an input operator, one or more dataprocessing sub-units connected in a sequence, and a sequence outputoperator. The upstream end of the sequence of data processing sub-unitscan be connected to the input operator and the downstream end of thesequence of data processing sub-units can be connected to the outputoperator.

Configuring the data stream processing system 200 b using a plurality ofdata processing segments 240 can simplify management of system 200 b.Each data processing segment 240 can be associated with one or moreprocessing tasks. When a set of processing tasks need to be performedfor downstream applications, the data processing segment 240 thatperforms some or all of the processing tasks in that set can be added tothe system 200 b as a unit. Similarly, where further processingbandwidth is required, an additional data processing segment can beadded in parallel to increase the volume of data that can undergo theparticular processing tasks in a given time period.

Additionally, the input operators and output operators in the dataprocessing segments 240 can be configured to provide connectivity withminimal initialization required. For instance, the input operators andoutput operators may be configured to communicate using UDP so that dataprocessing segments 240 can be inserted into a data stream processingsystem 200 b and begin communicating without any initial handshakingrequired.

As shown in the example of FIG. 2B, the stream processing system 200 bis connected to two separate incoming data streams, 111 a and 111 b. Thedata from the first incoming data stream 111 a can be routed to dataprocessing block 240 a while the second incoming data stream 111 b canbe routed to data processing block 240 b. In some cases, the dataprocessing blocks 240 a and 240 b may perform different processingoperations on the data streams 111 a and 111 b respectively. Forinstance, the format or content of the data in streams 111 a and 111 bmay be different and may thus require different processing to beperformed.

In some embodiments, the stream processing system 200 b may include aplurality of parallel data processing segments. Each parallel dataprocessing segment in the plurality of parallel data processing segmentscan include an instance of the same sequence input operator, an instanceof the same data processing sequence, and an instance of the samesequence output operator.

For example, data processing segments 240 a and 240 b may each includethe same sequence of input operator, one or more data processingsub-units, and output operator. The data processing segments 240 a and240 b may thus provide the same processing functionality as one another.This may allow throughput of system 200 b to be managed and potentiallyincreased by increasing the volume of data that can be processedconcurrently. Providing processing blocks in parallel can allow scalingof operators as well as processing data units.

As shown in FIG. 2B, the output data stream from processing blocks 240 aand 240 b can be routed to processing block 240 c. The processing block240 c can perform a defined sequence of operations on the data receivedfrom both processing blocks 240 a and 240 b.

In some embodiments, the data processing sub-units within the dataprocessing blocks, such as data processing block 240 c, may include aplurality of parallel sub-unit instances. Each parallel sub-unitinstance can be configured to modify the data in the received datastream in the same manner. Additionally, each of the parallel sub-unitinstances can be connected to the same preceding data processingsub-unit.

One or more distributive compositional operators can be connected to,and upstream from, the parallel sub-unit instances. Distributiveoperators may distribute the outgoing data stream (i.e. the data streamsreceived from processing blocks 240 a and 240 b) between the parallelprocessing sub-units. Examples of distributive operators can includeload-balancing operators and conditional operators such as those shownin FIGS. 3B and 3C described herein below.

Once data has been modified by the data processing block 240 c, theoutgoing data stream can be directed to processing blocks 240 d and 240e. The data processing block 240 c may include a distributive and/orduplicative compositional operator usable to route data to theprocessing blocks 240 d and 240 e.

For instance, in some cases the outgoing data stream may be selectivelydistributed to one or both of processing blocks 240 d and 240 e. Inother cases, the outgoing data stream may be duplicated and sent to bothprocessing blocks 240 d and 240 e.

In the example shown in FIG. 2B, the processing blocks 240 d and 240 eare connected to separate outgoing data streams 121 a and 121 b. Each ofthe outgoing data streams 121 a and 121 b can be connected to a dataoutput unit such as monitoring and/or analysis applications and/or datastorage applications. The monitoring and/or analysis applications can beconfigured to operate on the processed data in the data streams 121 aand 121 b to identify events of interest from the incoming data streams111 a and 111 b. The data storage applications can be configured tostore the processed data in the data streams 121 a and 121 b innon-volatile storage memory. The data storage applications may store theprocessed data for later retrieval and/or analysis. The processed datamay be stored with time index data determined by the precedingprocessing blocks (e.g. time index data may be inserted into headers ofprocessed data items). The time index data may reflect the time at whicheach data item was generated and/or received by the system 200 b. Thismay facilitate later analysis, by permitting a sequence of the storedprocessed data to be determined.

In some cases, each output data stream 121 a and 121 b can include datafrom one or both of the input data streams 111 a and 111 b. In someexamples, output data stream 121 a may receive processed data from onlyone of the input data streams 111 a and 111 b, while output data stream121 b receives processing data from the other input data stream.

Alternatively, the processed data that is routed to the output datastream 121 a or the output data stream 121 b may depend on theinformational content of the data stream received from the input datastreams 111 a and 111 b. For instance, the processing block 240 c mayinclude compositional operators configured to control the data that isdirected to the processing blocks 240 d and 240 e, and in turn outputdata streams 121 a and 121 b.

As shown, the separate output data streams 121 a and 121 b can share atleast one processing segment 240 c. This may be the case even where theoutput data streams 121 a and 121 b receive processed data correspondingto entirely different input streams 111 or entirely differentinformational content. For instance, the downstream output unit (e.g.monitoring applications) connected to output streams 121 a and 121 b mayrequire some of the same processing operations to be performed (e.g. thesame enrichment data added or the same file format of data). By sharinga processing segment 240 c, the size and resource utilization of system200 b may be reduced.

Referring now to FIG. 2C, shown therein is a flowchart illustrating amethod or process 250 of processing streaming data. Method 250 may becarried out by various components of system 100 such as the data streamprocessing system 105 and data stream processing systems 200 a and 200b.

At 255, at least one incoming data stream can be received. Each incomingdata stream can be received from a data source that provides datarelevant to a system or network that is being monitored. For example,the incoming data streams can include various different types of datasuch as network flow data (NetFlow, IPFix, sFlow, Bro Flows, etc.),syslog event data, DNS request logs, host event logs, collected threatinformation, and data from network probes, Internet of Thingsapplications, and other data generating devices.

At 260, the data from each incoming data stream can be routed through adata processing graph. The data processing graph can be defined toinclude a plurality of data processing sub-units and a plurality ofoperators that connect the data processing sub-units. The operators canbe arranged to define the route through the data processing graph foreach receiving data stream.

The processing graph can be defined as a feedforward or directed acyclicgraph. The data received from the incoming data streams can be routedunidirectionally towards one or more processed data outputs. Eachprocessed data output can be connected to a data output unit. Each dataoutput unit may include a data analysis application and/or data storageapplication. The data from the incoming data streams can be routedthrough the processing graph to one or more of the downstream dataoutput applications (e.g. data processing applications and/or datastorage applications) to provide those applications with the requiredpre-processed data.

The operators used to define the processing graph can be configured todirect data streams to the various processing sub-units. Some of theoperators can be configured to transmit and/or receive data using theUser Datagram Protocol (UDP). Similarly, some of the data processingsub-units can be configured to transmit and/or receive data using UDP.This may facilitate on-the-fly changes to the data processing graph.This may also provide high throughput data transmission, ascommunication initialization procedures may be avoided.

At 265, the data in the data streams can be modified by the dataprocessing sub-unit. Each data processing sub-unit can receive anincoming stream of data and perform one or more operations on thereceived data. The data processing sub-unit can then output a modifieddata stream that results from the operations performed.

As data is routed through the processing graph, it may be directedthrough multiple data processing sub-units. As a result, multipleprocessing operations may be performed on the data in each data stream.The particular types and sequence of data processing units (andcorresponding data processing operations) may vary depending on therequirements of the downstream data output units (e.g. data analysisapplications and/or data storage applications).

At 270, the data stream processing system can output one or moreprocessed data streams. The processed data streams correspond to theincoming data streams after having been modified by the processingsub-units through which they have been routed.

At 275, a controller can monitor the processing requirements of the dataprocessing graph. For example, the controller may monitor theapplications that are currently coupled to the data processing graph. Bymonitoring the applications that are currently coupled to the dataprocessing graph, the controller may identify the necessary dataprocessing steps that need to be performed by the data processing graphbased on the type of processed data required by those applications inorder to operate.

The controller can also identify changes in the processing requirements,for instance as the result of changes to the applications connected tothe processing graph or changes to the data sources. The controller mayreceive a request for a new processing application to be deployed. Thecontroller may then determine that modifications to the processing graphmay be required in order to route data receive at 255 through theprocessing sub-units necessary for the new application.

The controller may also determine that modifications to the processinggraph may occur when an application is removed or no longer connected tothe data stream processing system.

The controller can also monitor the current status of the variousoperators and processing sub-units in the processing graph. Thecontroller can identify various performance deficiencies in theoperators and/or processing sub-units that may indicate that updates tothe data processing graph may be required.

For instance, the controller can monitor performance characteristics ofeach of the data processing sub-units. Sub-unit performancecharacteristics can include characteristics such as the input date rate,output data rate, buffer fill etc. The controller may identify aperformance deficiency or performance issue in one or more dataprocessing sub-units, such as load imbalances, input data overloads, lowoutput data rates, sub-unit faults etc.

The controller may similarly monitor the performance characteristics ofeach of the operators in the processing graph. The controller mayidentify a performance deficiency or performance issue in one or moreoperators in a manner analogous to the data processing sub-units.

At 280, the controller can update the data processing graph. Thecontroller may update the data processing graph in response tomonitoring the processing graph performance at 275. The data streamprocessing system may then continue to route data through the processinggraph at 260. Examples processes for updating the data processing graphare shown in FIGS. 5B and 5C and described in further detail hereinbelow.

Updating the data processing graph may involve adding and/or removingdata processing sub-units and/or operators. In some cases, thecontroller may update the processing system by scaling the dataprocessing sub-units to account for performance issues identified at275. Additional instances of data processing sub-units may be deployedto increase the resources available for a given processing operation.The control system may also update the interconnection of the operatorsto ensure that data is routed to the newly deployed data processingsub-unit instance.

For example, where the buffer fill of a given data processing sub-unitis determined to exceed a usage threshold, the controller may deploy anadditional instance of that data processing sub-unit. The dataprocessing graph may then be updated to re-route some data through thenewly deployed data processing sub-unit to alleviate the strain on theexisting data processing sub-unit.

In other cases, the controller may determine that new or differentprocessing sub-units are needed. For example, a new downstreamapplication may require different processing operations than thoseavailable in the existing processing graph. The processing graph maythen be updated by inserting instances of processing sub-unitsconfigured to perform the required processing. The operators in theprocessing graph can also be updated to route data through these newlydeployed processing sub-units.

The steps shown in method 250 are merely examples and it should beapparent that more or different method steps may be involved.Additionally, steps 255-280 may be performed concurrently and on anongoing basis while the data stream processing system is operational.

Referring now to FIGS. 3A-3D, shown therein are examples of varioustypes of compositional operators. Compositional operators can be used todefine the required processing subgraph for each downstream applicationby directing the appropriate data streams to the required dataprocessing sub-units. The compositional operators can be configured tocontrol what data is transmitted to downstream data processingsub-graphs. In some cases, the compositional operators may prevent datafrom being passed to downstream processing subgraphs and/or allow thedata to be transmitted to a subset of the downstream processingsub-graphs. The compositional operators can also be used to combine datafrom a plurality of different data streams. The compositional operatorsmay also be used to distribute or duplicate data streams.

For example, the data stream processing system may define a processinggraph that includes a plurality of parallel data processing sub-units.In some cases, the parallel sub-units may include parallel instances ofthe same data processing sub-unit. In such cases, each of the parallelprocessing sub-units can be configured to modify data in the receiveddata stream in the same manner. The compositional operators may thendirect data to one of these parallel data processing sub-units.

In some cases, the parallel data processing sub-units may not performthe same processing operations. Rather, the parallel data processingsub-units may provide different processing operations and thus modifythe received data in a different manner. The compositional operators maydirect data to one or both of these parallel data processing sub-units.

FIG. 3A shows an example of a duplicator operator 310. The duplicatoroperator 310 is an example of a duplicative compositional operator.

The duplicator operator 310 can be configured to receive an input datastream 311 and replicate the input data stream 311 into a plurality ofoutput data streams 312. Various configurations of the duplicatoroperator 310 may be implemented. In general, the duplicator operator 310can receive a single input data stream 311 and output multiple streams312 that include the data from that input data stream 311.

As shown in FIG. 3A, the duplicator operator 310 can be coupled to aplurality of parallel data processing sub-units 313 a-313 c. Each of theparallel data processing sub-units 313 may also be connected to the samepreceding data processing sub-unit(s) in the processing graph via theduplicator operator 310. The duplicator operator 310 can transmitmultiple copies of the input data stream that it receives, with one copyto each of the data processing sub-units 313 a-313 c.

The data processing sub-units 313 a-313 c may be configured to performdifferent processing operations. Accordingly, the data processingsub-units 313 a-313 c may correspond to a data processing sub-graphportion for different downstream analysis applications. For example, thedata processing sub-units 313 a-313 c may each perform different featureextraction operations required by different analysis applications.

The input data stream 311 to the duplicator 310 may be provided as asynchronous pipe-based input. For instance, the duplicator 310 mayreceive the input data stream in a First-In-First-Out (FIFO) arrangementwith a small buffer. In some embodiments, the duplicator 310 may alsooutput synchronous data using pipes for each of the output data streams312.

In other cases, the duplicator 310 may transmit the output data streams312 using UDP. For instance, UDP multicast may be used to replicate theinput stream 311 and transmit multiple output streams 312.

FIG. 3B shows an example of a load balancer operator 320. The loadbalancer operator 320 can be configured to receive an input data stream321 and distribute the input data stream 321 amongst a plurality ofoutput data streams 322. The load balancer operator 320 is an example ofa distributive compositional operator that distributes data from theinput data stream 321 into a plurality of separate output data streams322. The load balancer operator 320 may transmit all of the data fromthe input data stream 321 to downstream data processing sub-units 323.

As shown in FIG. 3B, the load balancer operator 320 can be coupled to aplurality of parallel data processing sub-units 323 a-323 c. Each of theparallel data processing sub-units 323 may also be connected to the samepreceding data processing sub-unit(s) in the processing graph via theload balancer operator 320. The load balancer operator 320 can transmitseparate portions of the input data stream that it receives to each ofthe data processing sub-units 313 a-313 c.

Various configurations of the load balancer operator 320 may beimplemented. In general, the load balancer operator 320 can receive asingle input data stream 321 and distribute the data from the input datastream 321 to the downstream data processing sub-units 323 a-323 c inorder to balance the data load on the data processing sub-units 323a-323 c. For example, the load balancer operator 320 may distribute datafrom the input data stream 321 proportionally amongst the dataprocessing sub-units 323 a-323 c. The load balancer operator 320 may usevarious load-balancing techniques, such as round robin load balancing orhash-based load balancing for example.

In some cases, the load balancer operator 320 may distribute the dataamongst the data processing sub-units 323 a-323 c based on theperformance of the data processing sub-units 323 a-323 c. For instance,the load balancer operator 320 may distribute data amongst the dataprocessing sub-units 323 a-323 c to maintain a substantially consistentor even buffer fill for the data processing sub-units 323.

In various embodiments, the load balancer operator 320 may be configuredto output data in various formats. In some examples, load balanceroperator 320 may be configured to transmit data to the data processingsub-units 323 using a pipe-based transmission schema. The load balanceroperator 320 may even use TCP if the system has sufficient bandwidth.

In some examples, the load balancer operator 320 may transmit data tothe data processing sub-units 323 using UDP. This may facilitate scalingthe data processing sub-units 323 while the system is operational, byadding or removing data processing sub-units, while the load balanceroperator 320 continues to transmit data.

Referring now to FIG. 3C, shown therein is an example of a conditionaloperator 330. The conditional operator 330 is an example of adistributive compositional operator that can receive an input datastream and distribute the data in the input data stream amongst aplurality of downstream processing sub-units 333 a-333 b. In some cases,the conditional operator 330 may also operate as a duplicative, orpartially duplicative, compositional operator.

The conditional operator 330 can be configured to receive an input datastream 331 and distribute the data in the input data stream 331 to oneor more output data streams 332 based on whether the data satisfies aset of conditions. The conditional operator 330 can define a set ofconditions associated with each of the output data streams 332. Theconditional operator 330 can then determine, for each block of data(e.g. file) in the input data stream 331 whether that block of datasatisfies defined conditions. The conditional operator 330 can thentransmit the data to the corresponding one or more output data stream332 or prevent the data from passing to the output data streams 332based on whether the set of conditions is met.

For example, the conditional operator 330 may route a block of data toone of the data processing sub-units 333 a or 333 b based on the blockof data satisfying a given condition. For instance, a block of data maybe routed to data processing sub-units 333 a or 333 b according towhether it pertains to an IP address that is of ingress or egress typefor a given network. The conditional operator 330 may be used to scalethroughput in the downstream data processing sub-units 333 a and 333 b.

In some cases, the conditional operator 330 may be used to filter datafor downstream processing sub-units 333. For instance, the downstreamprocessing sub-units 333 a and 333 b may correspond to differentprocessing operations required for different storage and/or analysisapplications. These storage and/or analysis applications may requireincoming data having different initial characteristics, or they maytarget different aspects of an incoming data stream. The conditionaloperator 330 may then direct the appropriate data to the appropriatedata processing sub-unit 333 a or 333 b using the set of conditions todetermine whether the block of data has characteristics appropriate forone or both of the downstream storage and/or analysis applications.

Referring now to FIG. 3D, shown therein is an example of a combineroperator 340. The combiner operator 340 receives a plurality of incomingdata streams 341 from a plurality of upstream data processing sub-units343 a-343 c. The combiner operator 340 can then merge the incoming datastreams 341 into a single output stream 342. The combiner operator 340may combine the data in the incoming data streams 341 without regard toordering of the data.

In the example shown, the combiner operator 340 is configured to joinUDP data streams into a synchronous pipe data stream. Each incoming datastream 341 is received using the UDP transmission protocol. The combineroperator 340 can output the received data using a pipe-basedcommunication protocol. The combiner operator 340 is an example of acompositional operator that may also operate as a connectivity operatorin that it provides connectivity between processing segments usingdifferent communication protocols. Various other configurations of acombiner operator 340 may be implemented, including some that may notmodify the communication protocol used to transmit data.

In general, the compositional operators 310, 320, 330 and 340 can bescaled while the data stream processing system is operational. This mayallow the data stream processing system to update and modify theallocation of resources as new applications or new data sources areadded to the data stream processing system. In various embodiments, thecompositional operators 310, 320, 330 and 340 may be implemented ashigh-performance code on general purpose computing and/or specialpurpose hardware (FPGAs, GPUs, ASICs, special purpose processors).

Referring now to FIGS. 4A-4E, shown therein are various examples ofconnectivity operators. Connectivity operators can be used to providethe required connectivity between subsequent data processing sub-unitsin a data processing graph. The connectivity operators can be configuredto modify the communication protocol being used to transmit a stream ofdata to account for the requirements of a given data processingsub-unit.

As explained above, the operators used in the data stream processingsystems described herein may transmit data using various differentcommunication protocols. Some of the operators can be configured totransmit data using UDP. This may provide flexibility in run timechanges in services. UDP can also provide high throughput data transferfor systems where the data processing sub-units have sufficientbandwidth (and sufficient buffer space) to process all the receiveddata.

Operators can also be configured to use other communication protocols totransmit data. For example, TCP or pipe-based communication protocolsmay be used to provide additional data transfer reliability and ensurethat a data processing sub-unit receives the appropriate blocks of datarequired for the operations to be performed.

Referring now to FIG. 4A, shown therein is an example of a Pipe-to-UDPconnectivity operator 410. Operator 410 can receive an incoming datastream 411 that is pipe-based and convert the data stream 411 to anoutgoing UDP stream 412. The synchronous incoming data stream 411 canthus be converted to an asynchronous outgoing data stream 412. ThePipe-to-UDP operator 410 can also be configured to transmit data usingUDP multicast to provide further flexibility in data output andtransmission.

The incoming data stream 411 may correspond to the output data streamfrom a data processing sub-unit. The operator 410 may then route theoutput data stream from that data processing sub-unit to a downstreamdata processing sub-unit. By positioning the operator 410 to receive theoutput data stream from a data processing sub-unit, and then re-transmitthe output data stream using UDP, the output data stream can be easilyre-directed to a different downstream data processing sub-unit when thedata processing graph is modified or updated. The operator 410 need notperform any initialization procedures prior to re-routing the outgoingdata stream 412, which can facilitate updating the processing graph inreal-time.

Referring now to FIG. 4B, shown therein is an example of a UDP-to-Pipeconnectivity operator 420. Operator 420 can receive an incoming datastream 421 in UDP and convert the data stream 421 to an outgoingpipe-based stream 422. The asynchronous incoming data stream 421 canthus be converted to a synchronous outgoing data stream 422. Theoperator 420 can listen to a corresponding UDP socket and write the datait receives to a pipe 422.

The Pipe-to-UDP operators 410 and UDP-to-Pipe operators 420 can be usedto introduce parallel instances of a data processing sub-unit and thusscale the total throughput of the data processing sub-unit. ThePipe-to-UDP operator 410 and UDP-to-Pipe operator 420 may facilitateestablishing loose couplings between consecutive data processingsub-units or data processing sub-unit segments. This may allow data tobe redirected through newly deployed data processing sub-units on thefly.

For example, a data processing segment (such as segments 240 shown inFIG. 2B) can be configured with a UDP-to-Pipe operator 420 as the inputoperator and a Pipe-to-UDP operator 410 as the output operator. Thesequence of data processing sub-units within the segment may thencommunication data using a pipe to facilitate data reliability.Additionally, the input operator and output operator can be easilyadjusted to change where the incoming data stream is coming from andwhere the outgoing data stream is going to because the data is beingtransmitted using UDP. Additionally, newly deployed data processingsub-units can be connected using the multicast functionality of thePipe-to-UDP operator 410.

For example, N parallel Pipe-to-UDP multicast connector operators maydirect their data to M UDP-to-Pipe connector operators to create a databus. In another example, N parallel Pipe-to-UDP unicast connectoroperators may direct their data to a single UDP-to-pipe connectoroperator to create a join.

Referring now to FIG. 4C, shown therein is an example of a Pipe-to-TCPconnectivity operator 430. Operator 430 can receive an incoming datastream 431 that is pipe-based and convert the data stream 431 to anoutgoing TCP stream 432. The Pipe-to-TCP operator 430 may providereliable server-to-server transfer when data is being transmittedthrough multiple links involving multiple switches and buffers.Pipe-based communication may omit buffering. Accordingly, converting thedata transmission to TCP may facilitate the transmission of data overlonger distances, such as between remote servers.

Referring now to FIG. 4D, shown therein is an example of a TCP-to-Pipeconnectivity operator 440. Operator 440 can receive an incoming datastream 441 in TCP and convert the data stream 441 to an outgoingpipe-based stream 442. The operator 440 can listen to a correspondingTCP socket and write the data it receives to a pipe 442. Operator 440can convert data from using TCP transfer with buffering to pipe-basedtransfer that can omit buffers. The operator 440, in conjunction withoperator 430, may be usable to provide reliable (i.e. low loss) transferof data streams.

Referring now to FIG. 4E, shown therein is an example of a pipe-switchoperator 450. The pipe-switch operator 450 can be used to adjust orswitch the downstream destination of data from an incoming data stream451 between two different output data streams 452 and 453. The pipeswitch 450 may be used to switch the outgoing data stream betweendifferent data processing sub-units 460 and 461. For instance, wheredata processing sub-unit 460 is being updated or replaced by dataprocessing sub-unit 461, the pipe-switch operator 450 can redirect theflow of data from data processing sub-unit 460 to data processingsub-unit 461 in real-time during runtime without interrupting the dataflow. This may allow, for instance, the set of templates or enrichmentdata used by data processing sub-unit 460 to be replaced while datacontinues to be transmitted and processed.

In some cases, a pipe switch 450 may be positioned in a processing graphupstream from a data processing sub-unit that may be expected to bemodified or updated. For example, a pipe switch 450 may be positionedupstream from an enrichment processing sub-unit that may be updated ormodified frequently. This may facilitate rapid replacement and updatingof the enrichment processing sub-unit.

Referring now to FIG. 5A, shown therein is an example of a controlsystem 500. The control system 500 may be used in implementations ofdata processing system 100 and data stream processing systems 105, 200 aand 200 b. The control system 500 may also be referred to herein as acontroller and/or system manager.

The controller 500 can be used to manage the configuration of thecomponents within a data stream processing system. The controller 500can manage the deployment of data processing sub-units and operator. Thecontroller 500 can also manage the connections between the operators anddata processing sub-units to define a data processing graph. Thecontroller 500 may also be used to monitor the status and processingrequirements for the processing graph of the data stream processingsystem and modify the processing graph as required.

The controller 500 can be coupled to the plurality of data processingsub-units and the plurality of operators in the data processing graph.The controller 500 can also be coupled to the data output units (e.g.data storage and/or analysis applications) that receive data streamsoutput from the data processing graph. The controller 500 can also beconnected to the input of the data stream processing system to identifythe incoming data streams connected thereto.

The controller 500 can include a data stream processing (DSP) manager505 and a processing graph controller 520. The DSP manager 505 andprocessing graph controller 520 can be used to ensure that the datastream processing system is operating reliably and to provide theprocessing required for downstream analysis applications.

The processing graph controller 520 can be used to monitor the statusand performance of the data processing sub-units and operators that arecurrently deployed. The processing graph controller 520 can communicatewith a processing sub-unit controller 510 and an operator controller515. The sub-unit controller 510 and, in turn, monitor the status andperformance of the deployed data processing sub-units using sub-unitcontrol agents 525. Similarly, the operator controller 515 can monitorthe status and performance of the deployed operators using operatorcontrol agents 530.

The sub-unit control agents 525 and operator control agents 530 maycorrespond to the sub-unit monitoring agents 221 and operator monitoringagents 211 shown in FIG. 2A and described herein above. In general, thesub-unit control agents 525 and operator control agents 530 can monitorvarious performance characteristics, such as memory, CPU, bandwidthusage, throughput, input data rate, output data rate etc. for the dataprocessing sub-units and operators connected thereto.

The sub-unit controller 510 and operator controller 515 may then collectthe monitoring data from all of the data processing sub-units andoperator sub-units and provide this monitoring data to the processinggraph controller 520. The processing graph controller 520 may thendetermine the status and processing capability of the data processinggraph as a whole.

The DSP manager 505 can be configured to determine whether modificationsto the data processing graph are required. The DSP manager 505 canmonitor the processing requirements for the data processing graph basedon the data output units (e.g. data storage and/or analysisapplications) and incoming data streams coupled to the data processinggraph.

For example, the DSP manager 505 can receive requests for new ordifferent data output units (e.g. data storage and/or analysisapplications) to be connected to the data stream processing system. TheDSP manager 505 may then update the processing requirements for the datastream processing graph based on the data required by the data outputunits (e.g. data storage and/or analysis application) to function.Similarly, where applications are removed and thus no longer requireoutput from the data stream processing system, the DSP manager 505 canupdate the processing requirements accordingly.

The DSP manager 505 can also monitor the number and type of incomingdata streams connected to the data stream processing system. When thereare changes to the incoming data streams, the DSP manager 505 can updatethe processing requirements of the data stream processing system (e.g.to indicate that a new incoming data stream needs to be processed).

The DSP manager 505 can also determine whether the data processing graphrequires any updates or modifications. The DSP manager 505 may comparethe determined processing requirements to the processing capabilitiesidentified by the processing graph controller 520. The DSP manager 505may then determine whether there is a mismatch between the existingprocessing requirements and the processing capabilities.

The DSP manager 505 may identify updates that are required to the dataprocessing graph to account for the mismatch in the data processingrequirements and capabilities. For example, the DSP manager 505 mayidentify new and/or different data processing sub-units and/or operatorsthat can be deployed to provide the necessary processing functionality.The DSP manager 505 may then modify the processing graph by deployingnew and/or different data processing sub-units and or operators,updating the connections between operators and data processingsub-units, and/or redirecting data that is transmitted by one or moreoperators. For instance, the DSP manager 505 may transmit instructionsto the sub-unit controller 510 and/or operator controller 515 to modifythe configuration of the data processing sub-units and/or operators inthe data processing graph. The DSP manager 505 can modify the processinggraph while the system is operational, for instance using operatorsconfigured to transmit data using UDP.

Although shown as different elements, it should be understood that thefunctions of the DSP manager 505, processing graph controller 520,processing sub-unit controller 510 and operator controller 515 may forinstance be implemented as a combined management controller.

In some embodiments, the controller 500 may operate autonomously, orsemi-autonomously to define and update the data processing graph. Forexample, the controller 500 may receive application requests, determineany modifications required to provide the requested application with thenecessary processed data, and automatically update the data processinggraph to provide the required processing functionality. An example of aprocess for modifying the data processing graph in response to anapplication request is shown in FIG. 5B.

Similarly, the controller 500 may monitor the operation of the dataprocessing graph and determine that the data processing graph is notproviding the desired processing functionality, or performance is aboutto degrade, based on the monitored status and/or performance of the dataprocessing graph. The controller 500 may then automatically update thedata processing graph to ensure that the data processing graph continuesto provide the required processing functionality. An example of aprocess for modifying the data processing graph based on monitoring ofthe processing graph is shown in FIG. 5C.

In some embodiments, the controller 500 may be coupled to anadministrator computer. The controller 500 may transmit notifications tothe administrator computer such as alerts regarding performance issuesor additional processing requirements. A user of the administratorcomputer may then direct the controller 500 to implement modificationsto the data processing graph to account for the performance issuesand/or changes to processing requirements.

In some instances, the controller 500 may also identify the requiredmodifications and then present those modifications for approval by theuser of the administrator computer.

In some embodiments, the controller 500 may operate to update the dataprocessing graph automatically within certain defined constraints. Forinstance, an administrator may define a range of resources that thecontroller 500 can access automatically. In such embodiments, if thecontroller 500 determines that additional resources may be required toprovide the required processing functionality, the controller 500 maythen transmit an alert to the administrator computer indicating thatapproval for additional resources may be required. A user of theadministrator computer may then approve or deny the request foradditional resources.

In some embodiments, the operation of the controller 500 may befacilitated using a language to describe the processing graph of dataprocessing sub-units and operators. The language may be used in a userinterface provided to an administrator computer in communication withcontroller 500. A user of the administrator computer can use thelanguage to define new application request 635.

The language may facilitate the orchestration and deployment of new dataprocessing sub-units and operators. The language can be used by graphcontroller 520 to determine what adaptation actions are required, e.g.scaling of resources and/or rate limiting at the ingress to the overallsystem.

For example, Python may be used to define a processing graph at a highlevel by specifying the data processing sub-units, their configurations,and their interconnections using operators. Python scripts may then beused to generate a JSON representation of the graph that can be passedto other modules, such as for control, monitoring, scaling, deployment,and drawing. FIG. 7C is an example of a visual representation of aprocessing graph generated in this manner.

Referring now to FIG. 5B, shown therein is an example process 550 forupdating a data processing graph. Update process 550 is an example of aprocess that may be used to update a data processing graph in responseto a change in the analysis applications or incoming data streamsconnected to a data stream processing system. Process 550 may be used invarious embodiments of data processing system 100 and data streamprocessing systems 105, 200 a and 200 b.

At 552, a stream processing request can be received. For example, theDSP manager 505 may receive a request to support a new data output unit(e.g. data storage and/or data analysis application) or a new incomingdata stream. The request may identify a new data output unit (e.g. datastorage and/or data analysis application) or data source to be connectedto the data stream processing system.

In some cases, the DSP manager 505 may receive a processing request froma data output unit (e.g. data storage and/or data analysis application)that is already connected to the data stream processing system. Forinstance, the data output unit (e.g. data storage and/or data analysisapplication) may have been updated and thus requires differentprocessing functionality.

The DSP manager 505 may determine, based on the received requests, theprocessing functionality required to implement the requested datastorage and/or analysis application or data source.

In some cases, the DSP manager 505 may also determine that a data outputunit (e.g. data storage and/or data analysis application) or data sourceis no longer being used (and thus no longer requires processed data fromthe data stream processing system). The DSP manager 505 may thendetermine updated processing requirements for the data streamingprocessing system based on the remaining data output units (e.g. datastorage and/or data analysis applications) and data sources.

At 554, the controller 500 can determine whether the processing graph isconfigured to satisfy the processing request. The DSP manager 505 maycompare the existing processing functionality, e.g. using performanceand status data on the deployed data processing sub-units and operatorsfrom the processing graph controller 520, with the updated processingrequirements.

The DSP manager 505 may determine whether the existing processing statusof the data stream processing system (using status data from theprocessing graph controller 520) can satisfy the processingfunctionality required by the change in data output units (e.g. datastorage and/or data analysis applications) and/or data sourcesidentified at 552. The DSP manager 505 may also determine whether thenew processing requirements may result in adverse effects on theexisting processing system (e.g. introducing additional data sources mayincrease the volume of data being processed, which may increase the dataprocessing bandwidth required). Upon determining that the processinggraph is not configured to satisfy the processing request, the DSPmanager can modify the processing graph to enable the processing graphto satisfy the processing request.

In some cases, the DSP manager 505 may determine that little to nomodifications are required to the data stream processing system. Forinstance, the DSP manager 505 may determine that an existing outgoingdata stream can provide a new data output unit (e.g. data storage and/ordata analysis application) with the required processing data. The DSPmanager 505 may then simply couple the new data output unit (e.g. datastorage and/or data analysis application) to that outgoing data streamas well.

Similarly, the DSP manager 505 may determine that an incoming datasource can be processed in a manner analogous to an existing incomingdata stream. The new incoming data stream may then be directed to thesame processing sub-graph as the corresponding existing data stream. Insome cases, the DSP manager 505 may also determine that the existingprocessing sub-graph may require additional parallel processing segmentsand/or sub-units to account for the increased data volume. The DSPmanager 505 may then direct the deployment of the additional processinginstances (and any corresponding operators required).

In some cases, the DSP manager 505 may determine that a modification tothe data stream processing system can be made, but is not required. Forinstance, when a data output unit (e.g. data storage and/or dataanalysis application) or data source is no longer being connected theDSP manager 505 may determine that some data processing sub-units may beremoved. In some cases, the DSP manager 505 may then remove those dataprocessing sub-units from the data processing graph to reduce power andresource utilization.

If modifications are required to the data processing graph, the DSPmanager 505 can determine an updated data processing graph required toprovide the new processing functionality. The DSP manager 505 mayidentify the overall data processing graph required for the newprocessing functionality and then the modifications required to providethe updated data processing graph.

The modifications to provide the updated data processing graph mayinvolve scaling existing data processing sub-units and/or modifying theconnections provided by existing operators. In some cases, themodifications may include deploying new operators and/or data processingsub-units. The DSP manager 505 can then manage the updates to the dataprocessing graph at 556 to 560. The DSP manager 505 may provideinstructions to the processing controller 510 and operator controller515 to implement the required modifications.

As shown in FIG. 5B, the steps 556-560 may be considered optional in thesense that not all steps need to be performed to update the processinggraph. In some cases, all of steps 556-560 may be performed to updatethe processing graph to provide the processing functionality identifiedat 552 and 554. However, in some embodiments updating the processinggraph may require only redirecting some operators or scaling ofprocessing sub-units along with associated redirection of operators forexample. Various other permutations may also be used depending on thespecific updates to the processing graph that are required.

At 556, one or more operators in the processing graph may be scaled. Toscale the operators, additional instances of one or more operators maybe positioned in parallel to increase the available bandwidth for theoperation performed by that given operator. For instance, multipleconnectivity operators may be deployed in parallel to increase the rateof data that can be converted from one communication protocol toanother. Similarly, multiple compositional operators may be deployed inparallel to increase the rate at which data is directed to theappropriate downstream processing sub-units.

Scaling the operators may also involve adjusting the connections betweenoperators and other operators or data processing sub-units. Forinstance, the parallel instances of an operator may be connected to thesame upstream operator or data processing sub-unit or the samedownstream operator or data processing sub-unit. Scaling operators maythus involve updating the connectivity to those existing operatorsand/or data processing sub-units.

At 558, one or more new operators and/or data processing sub-units canbe deployed. The DSP manager 505 may deploy new operators to supportupdating the connections between the processing sub-units in the dataprocessing graph. New data processing sub-units can be deployed tosupport new processing functions that are required as a result of thenew data source and/or data output unit (e.g. data storage and/or dataanalysis application). In some cases, additional instances of existingdata processing sub-units may be deployed in parallel to increase thebandwidth available for certain processing functions.

In some embodiments, newly deployed data processing sub-units mayprovide processing operations that are different from the existing dataprocessing sub-units in the data processing graph. This may occur, forinstance, where the data from the new data source has characteristicsrequiring new types of processing operations. This may also occur, forinstance, where the new data output units (e.g. data storage and/or dataanalysis applications) requires different features to be extracted ordifferent enrichment data.

In some embodiments, newly deployed data processing sub-units may beinstances of data processing sub-units already present in the dataprocessing graph. This may occur to provide parallel processingsub-units. In other cases, this may be done so that a differentprocessing sub-graph undergoes the processing operations for that dataprocessing sub-unit. Rather than redirecting data through an existingdata processing sub-unit, it may be simpler to add a new instance ofthat sub-unit.

In some embodiments, the operators and data processing sub-units can bedeployed as data processing segments. For instance, data processingsegments such as data processing segment 240, or those shown in FIGS.6A, 6B, and 7A-7C may be deployed as a collective segment. Collectively,the processing segment may provide processing functionality thatcorresponds to a particular analysis application or incoming datasources. This may facilitate updating the data processing graph as thedata processing segments for certain processing functions can be definedin advance and then deployed by DSP manager 505 as required, withoutneeding to determine a sequence of operators and processing sub-unitsafter a processing request is received.

Deploying data processing segments may also facilitate updating theconnections between the existing processing sub-units and operators andthe newly deployed segment. For instance, the data processing segmentscan be bookended by a connector input operator and connector outputoperator respectively suitable to connect to operators upstream anddownstream from where that data processing segment is to be deployed.This may ensure that connectivity to the new data processing segment canbe achieved during runtime while the system is operational.

For example, the data processing segment may include a UDP-to-Pipeconnector as the input operator and a Pipe-to-UDP connector as theoutput operator. This may allow the data processing segment toautomatically begin receiving data using UDP once deployed (and once theupstream and downstream operators/data processing sub-units areconnected thereto) and to ensure that the received data can be routedthrough the sequence of data processing sub-units appropriately.

At 560, the operators in the data processing sub-graph can be updated toredirect data to new and/or different operators or data processingsub-units. For instance, where a new data processing sub-unit isdeployed at 558, an operator upstream from that newly deployedprocessing sub-unit can redirect some, or all, of the data it istransmitting to that newly deployed processing sub-unit. Similarly,operators downstream from the newly deployed data processing sub-unitand/or operator can be configured to receive data from the output of thenewly deployed operators and/or data processing sub-units.

At 562, the new incoming data stream or outgoing data stream can beconnected to the data processing graph. The DSP manager 505 may alsoprovide instructions to the processing controller 510 and operatorcontroller 515 to synchronize the updates to the data processing graph,with the implementation of the new incoming or outgoing data stream.This can ensure that the required processing functionality is in placeprior to inserting a new data output unit (e.g. data storage and/or dataanalysis application) and/or data source.

A similar procedure may be followed where the DSP manager 505 determinesthat an updated version of an operator or data processing sub-unitexists and should be deployed. Using steps 556-560, the DSP manager 505can deploy the updated version of the operator or data processingsub-unit and update the data processing graph as required to redirectdata through the updated component.

Referring now to FIG. 5C, shown therein is an example process 570 forupdated a data processing graph. Update process 570 is an example of aprocess that may be used to update a data processing graph in responseto monitoring the status and/or performance of a data stream processingsystem. Process 570 may be used in various embodiments of dataprocessing system 100 and data stream processing systems 105, 200 a and200 b.

The DSP manager 505 can monitor the status and/or performance of thedata processing graph using collected data provided by processing graphcontroller 520. The processing graph controller 520 can be configured tomonitor the performance of each data processing sub-unit and eachoperator. For instance, CPU usage, memory usage, buffer usage andbandwidth for each data processing sub-unit and operator may bemonitored.

At 572, the DSP manager 505 can determine that a modification to thedata processing graph is required. The DSP manager 505 may determinethat one or more data processing sub-units and/or operators have reachedperformance thresholds indicating an undesired level of performance orthat an undesired level of performance is likely to occur.

The controller 500 may define various operational requirements for thedata stream processing system. For instance, each of the data processingsub-units and operators within a data processing sub-graph that leads toa given data output unit (e.g. data storage and/or data analysisapplication) may have associated operational requirements such aslatency requirements, loss requirements etc. The controller 500 may alsodefine monitoring and performance thresholds corresponding to theoperational requirements. The performance thresholds may be defined inrelation to the processing operations of a data processing sub-unit oroperator (e.g. throughput, input data rate, output data rate, loss rateetc.) and/or in relation to the status of the data processing sub-unitor operator (e.g. CPU usage, memory usage, buffer fill etc.). Theperformance thresholds can be defined to ensure that the operators anddata processing sub-units can provide the required processingfunctionality. Accordingly, the performance thresholds may be defined sothat modifications can be performed before unacceptable performancedegradations occur.

As explained herein above, the controller 500 can monitor the status andperformance of each operator and data processing sub-unit. Based on thismonitoring, the controller 500 can determine that one or more thresholdshas been reached. When a performance threshold has been reached, thecontroller 500 may determine that a modification to the data processinggraph is required to ensure that the operational requirements of thedata processing graph continue to be met.

Operator monitoring agents and data processing sub-unit monitoringagents can be configured to monitor the status of correspondingoperators or data processing sub-units. These monitoring agents cangenerate alerts when a performance threshold is reached or exceeded. Forexample, the monitoring agents may identify performance deficienciessuch as an input data rate overload, an output data rate being too low,and/or a subunit fault. In some cases, the processing graph controller520 may identify a performance threshold being reached based onmonitoring data from a plurality of monitoring agents (e.g. a loadimbalance). In some cases, the processing graph controller 520 may alsomonitor the input rate and output rate of the incoming and outgoing datastreams to identify potential performance deficiencies. For instance,the processing graph controller 520 may identify an increase in theincoming data volume that may overwhelm the capacity of the existingdata processing graph.

The processing graph controller 520 may then transmit to DSP manager 505an alert indicating that the performance threshold has been reached. DSPmanager 505 may then determine that a modification to the graph isrequired to ameliorate the performance deficiency or prevent theperformance deficiency from occurring.

At 574, the DSP manager 505 may determine what modifications may berequired to the data processing graph to avoid or remedy the performancedeficiencies identified at 572. The DSP manager 505 may determinewhether scaling of resources or rate limiting may be required to adaptto the changes in demand or resource capacity identified at 572.

In some cases, the DSP manager 505 may determine that the resources(e.g. virtual machines or containers) corresponding to the dataprocessing sub-units and/or operators need to be scaled or deployed toadapt to the performance issues identified at 572. The DSP manager 505may then modify the processing graph to response to the identifiedperformance deficiency.

At 576, the DSP manager 505 can identify the data processing sub-unitsthat can be scaled to account for the performance deficiency identifiedat 572. For instance, where the performance deficiency relates to a dataprocessing sub-unit that is nearing or reaching a buffer fill threshold,the DSP manager 505 may determine that additional instance of thatsub-unit should be deployed in parallel.

At 578, the DSP manager 505 can direct the deployment of the dataprocessing sub-units identified at 576. In general, the data processingsub-units may be deployed in a manner analogous to step 560 of method550. For instance, parallel instances of data processing sub-units maybe deployed to provide increased throughput for a given processingfunction. As with method 550, the data processing sub-units andoperators can be deployed as processing segments.

In some cases, data processing sub-units can be deployed to replaceexisting data processing sub-units. For instance, existing dataprocessing sub-units may be determined to be faulty or requiring anupdated. Functioning or updated data processing sub-units can bedeployed and the data can be redirected to the newly deployed processingunits. As at 562, the operators can be redirected to direct data to andfrom the newly deployed data processing sub-units.

At 580, operators in the data processing graph can be scaled to accountfor the performance deficiency. For example, additional operators may bedeployed to increase throughput as required. Similarly, the connectionsbetween the operators and other operators and/or data processingsub-units may be updated in a manner analogous to step 564 of method550.

In some cases, the DSP manager 505 may determine that admission controltechniques should be applied to reduce the volume of data beingprocessed. In such cases, the method 570 can proceed to step 582.

At 582, the DSP manager 505 can determine that the rate of data passingthrough the data stream processing is to be reduced. In some cases, theDSP manager 505 may then modify the operation of the input connection tolimit the rate of data entering the data stream processing system. Insome cases, the DSP manager 505 may throttle the rate of data enteringindividual operators and/or data processing sub-units.

The DSP manager 505 may transmit instructions to reduce the rate of dataentering the data stream processing system by dropping or sub-samplingthe data from the incoming data streams. The DSP manager 505 may drop orsubsample data entering the data stream processing system to account forsurges in data in one or more incoming data streams and/ordeficiencies/degradations in the available resources. This may ensurethat the data that enters the data stream processing system can beprocessed accurately and rapidly. In some cases, the DSP manager 505 maycontrol the volume of data entering the data stream processing systemtemporarily while the resources in the processing graph are scaled toaccount for performance deficiencies.

For example, the first data processing sub-unit that receives data froman incoming data stream may be configured to prevent data from enteringthe data stream processing system. The DSP manager 505 may transmitinstructions to that data processing sub-unit to drop packets/messagesfrom the incoming data to reduce the volume of data entering the system.By controlling the data entering the data stream processing system atthe input, the DSP manager 505 may ensure there is consistency in thedata processed throughout the data stream processing system (across thevarious processing sub-graphs). This may provide unbiased dropping ofdata so that no specific data processing sub-graph is impacted unevenly.

In some cases, individual operators and/or data processing sub-units maybe configured to rate limit their input data. The DSP manager 505 maydetermine that individual processing sub-units and/or operators requirea reduced volume rate and direct those components to limit the rate ofvolume entering those sub-units or operators. In such cases, theindividual operators and/or data processing sub-units may drop orsubsample messages/packets they receive. This may provide more selectivedata limiting to ensure that an increased volume of data is analyzedoverall. In some cases, rate limiting individual operators and/or dataprocessing sub-units may be performed temporarily while resources arescaled to account for the data processing deficiency identified at 572.

Referring now to FIGS. 6A-6B, shown therein are examples of dataprocessing segments that may be used with data stream processing systemssuch as data stream processing systems 105, 200 a and 200 b. FIGS. 6Aand 6B are examples of data processing segments that may be used toprocess large volumes of network monitoring data, such as NetFlow andsimilar network flow monitoring data.

FIG. 6A shows an initial data collection and parsing segment. A datacollection input 601 can provide an incoming data stream or streams ofnetwork monitoring messages. For instance, the incoming data stream mayinclude NetFlow (IPFIX) messages.

The incoming data stream 601 can be received by a load balancingoperator 600. The load balancing operator 600 can distribute the networkmonitoring messages 609 amongst a plurality of downstream collectorprocessing sub-units 610. In the example shown here, the load balancingoperator 600 transmits network monitoring messages 609 using UDP.

The collector processing sub-units 610 can be configured to generatenetwork flow records 611 from the received network monitoring messages609. The network flow records 611 may be augmented flow recordscorresponding to the receive network monitoring messages with additionalinformation added. For example, geolocation information may be added tothe network monitoring messages 609. In some cases, the collectorprocessing sub-units 610 may be implemented as nProbes.

The network flow records may be generated in a first format. Forexample, the network flow records may be generated using a JavaScriptObject Notation (JSON) format. The JSON format typically produces largefiles that may be bandwidth intensive.

The collector processing sub-units 610 can transmit the network flowrecords 611 to a corresponding plurality of parser sub-units 620. Insome embodiments, the collector processing sub-units 610 and parsersub-units 620 may communicate using TCP. This may increase thereliability of data transfer between the collector processing sub-units610 and the parser sub-units 620.

Each of the parser sub-units 620 can generate parsed network flowrecords 612 from the network flow records received from thecorresponding collector processing sub-unit 610. The parser sub-units620 may convert the network flow records from the first format to asecond format that is easier to transmit. The second format may have asmaller file size than the first format. For example, the parsersub-units 620 may generate the parsed flow records in a CSV file format.In other cases, the parser sub-units 620 may even generate parsed flowrecords in binary.

The plurality of parser sub-units 620 can transmit the parsed flowrecords 612 to a combiner operator 630. The combiner operator 630 canreceive the parsed flow records 612 from the plurality of parsersub-units 620 and combine them into a single data stream.

In the example shown, each parser sub-unit 620 is configured to transmitthe parsed network flow records using the User Datagram Protocol (UDP).The combiner operator 630 can be configured to receive the parsed flowrecords 612 from the parsers 620 using UDP and output a single datastream 631 as a pipe-based output. The combiner operator 630 may thuscombine the parsed network flow records 612 into a synchronous outputstream 631.

The number of collector processing sub-units 610 and parser sub-units620 in the data processing segment can be selected based on theaggregate rate of network monitoring messages arriving at 601. Dependingon the arrival rate of the network monitoring messages, a controller 500may adjust the number of collector processing sub-units 610 and parsersub-units 620.

The number of collector processing sub-units 610 and parser sub-units620 may be adjusted until the capacity of the load balancer 600 orcombiner 630 is reached. Once this capacity is reached, one or both ofthe load balancer 600 or combiner 630 may be scaled. In other cases, anadditional data processing segment may be implemented in parallel.

In some embodiments, the parsed network flow records 612/631 may requireenrichment data to facilitate downstream processing. FIG. 6B illustratesan example of a data processing segment that may be used to insertenrichment data into the parsed network flow records 612/631.

In general, at least one stream enrichment processing sub-unit 650 canbe coupled to the parsed network flow records 631. Each streamenrichment processing sub-unit 650 can be configured to generateenriched network flow records 651 by inserting enrichment data intoreceived parsed network flow records 631.

In embodiments of FIG. 6B, the stream enrichment processing sub-units650 can receive parsed network flow records (i.e. the informationcontent of the parsed network flow records) and insert additionalinformation to facilitate subsequent processing. The stream enrichmentprocessing sub-unit 650 may examine the values in the received datastream and add contextual information related to the received data. Forexample, an IP address included in a parsed network flow record may beenriched by adding the autonomous system that it belongs to, itsgeographic location such as its source country, and other information.The enrichment data may provide a richer set of data usable by adownstream analysis application.

In some embodiments, the enriched data stream may be provided to a datastorage application for storage. The stream enrichment processingsub-unit 650 may add time index data to the received data (e.g. inheaders of the processed data files). This may allow the data storageapplication to store the processed data stream in a time-ordered manner.This may facilitate subsequent access, playback, processing, andanalysis etc. In some cases, a compression sub-unit may also be includedprior to the data storage application to compress the data in the outputdata stream prior to storage.

In some cases, a switching operator may be positioned upstream from thestream enrichment processing sub-unit 650. The switching operator may beusable to direct the parsed network flow records between differentstream enrichment processing sub-units 650.

In some embodiments, a plurality of stream enrichment processingsub-units 650 may be positioned in parallel. The switching operatoroperable can be configured to direct the parsed network flow records toa subset of the stream enrichment sub-units in the plurality of streamenrichment sub-units. In some embodiments, a first subset of streamenrichment sub-units may correspond to a first set of enrichment dataand a second subset of stream enrichment sub-units may corresponds to adifferent set of enrichment data.

The switching operator may be configured to direct all of the parsednetwork flow records to a particular subset of the stream enrichmentsub-units. For instance, the switching operator may be used where theenrichment sub-units are being updated/replaced to redirect the parsednetwork flow records to the new/updated enrichment sub-units.

As shown in the example of FIG. 6B, the switching operator 640 can beimplemented as a pipe switch operator (see also FIG. 4E described hereinabove). The pipe switch operator may be used to redirect the parsednetwork flow records to a new/updated enrichment sub-unit 650.

In some cases, the switching operator may be conditional operator thatis configured to selectively direct parsed network flow records to aparticular subset of the stream enrichment processing sub-units.

In some cases, the switching operator may be omitted. For instance, adifferent enrichment processing sub-unit 650 may be deployed toreplace/update the existing enrichment processing sub-units.Alternatively, an operator may be deployed in real-time to redirect theparsed network flow records.

The enriched network flow records can then be output using a streamoutput operator 660. The stream output operator 650 may be configured tooutput the enriched network flow records as an asynchronous outputstream 661 using UDP.

In the example shown in FIG. 6B, a pipe switch operator 640 is connectedupstream from an enrichment processing sub-unit 650 that is in turnconnected to a Pipe-to-UDP operator 660. This may enable the enricheddata stream 651 to be transmitted as a UDP stream 661 to provide loosecoupling for the data processing segment. This may allow the dataprocessing segment to be flexibly deployed in a data stream processingsystem.

Referring now to FIG. 6C, shown therein is an example method 670 ofprocessing network flow monitoring data. Method 670 is an example of amethod for processing network flow monitoring data that may beimplemented using data stream processing system 105, 200 a and 200 b.For example, method 670 may be implemented using the data streamprocessing segments shown in FIGS. 6A and 6B.

At 672, at least one incoming data stream of network monitoring messagescan be received. For instance, the data stream may include NetFlowmessages.

At 674, the network monitoring messages can be distributed amongst aplurality of downstream processing units. The network monitoringmessages may be distributing to ensure that the processing load isbalanced amongst the downstream processing units. The number ofdownstream processing units may be selected to accommodate the arrivaldata rate of the network monitoring messages.

At 676, a plurality of network flow records can be generated from thenetwork monitoring messages. The network flow records may be generatedin a first format, such as a JSON format. The downstream processingunits can include a plurality of collected sub-units in parallel. Eachcollector sub-unit may generate network flow records based on theportion of the monitoring messages received at 674.

At 678, the network flow records can be transmitted to a plurality ofparser sub-units. Each collector sub-unit may have a correspondingdownstream parser sub-unit. The network flow records generated by thatcollector sub-unit can then be transmitted to the correspondingdownstream parser sub-unit.

The first format used to generate the network flow records may generatelarge files. Accordingly, the collector sub-units and parser sub-unitsmay communicate using TCP to provide increased reliability of datatransfer.

At 680, a plurality of parsed network flow records can be generated. Theparser sub-units may convert the network flow records from the firstformat to a second format. The second format may facilitate downstreamtransmission and analysis. The second format may generate files with asmaller file size than the first format. For instance, the second formatmay be a CSV format or binary format.

At 682, the plurality of parsed network records from the plurality ofparsers can be combined into a single output stream. This may facilitatetransmission to downstream storage, processing and/or analysiscomponents. For instance, the parsed network records may be combinedinto a synchronous output stream. In some cases, the parsed networkrecords may then be enriched with enrichment data that is selected basedon the information required by downstream storage and/or analysisapplications.

Referring now to FIGS. 7A and 7B, shown therein are examples of dataprocessing segment that may be used to extract features from receivednetwork flow records. For example, the data processing segments shown inFIGS. 7A and 7B may be positioned downstream from the data processingsegments shown in FIG. 6A or FIG. 6B to receive parsed network flowrecords or enriched network flow records respectively.

In general, the data processing segments shown in FIGS. 7A and 7B caninclude a duplicator operator that receives the network flow records.The duplicator operator can be coupled downstream from the combineroperator 630. In embodiments where the network flow records areenriched, the duplicator operator can be coupled downstream from theenrichment data processing sub-unit. The duplicator operator can beconfigured to replicate the received network flow records (e.g. parsednetwork flow records or enriched network flow records) and outputmultiple copies of the received network flow records.

A plurality of feature extraction processing sub-units can be coupled tothe duplicator operator. Each feature extraction processing sub-unit canbe configured to derive one or more network flow characteristics fromthe duplicated network flow records.

In some cases, such as embodiments of the data processing segment shownin FIG. 7A, the individual feature extraction processing sub-units mayreceive different network flow records. In other cases, such asembodiments of the data processing segment shown in FIG. 7B, theindividual feature extraction processing sub-units may receive the samenetwork flow records.

FIG. 7A shows a first example of a data processing segment that can beused to extract data features from network flow records. A stream ofparsed or enriched network flow records 702 can be received by thestream processing segment shown in FIG. 7A.

In some embodiments, the network flow records 702 may be transmittedusing UDP. In such embodiments, a UDP-to-pipe operator 705 can beimplemented to transmit the network flow records as a pipe-based stream704 as shown.

The network flow records can be directed to a duplicator operator 710.The duplicator operator 710 can be configured to replicate the receivednetwork flow records 704 and transmit a plurality of replicated networkflow record streams 706. Each replicated stream 706 may include all ofthe network flow records received at 704.

As shown in FIG. 7A, each replicated network flow record 706 can bedirected to a conditional operator 715. The conditional operators 715can control which of the extractors 720 the replicated network flowrecords 706 are directed to. Each conditional operator 715 can beconfigured to selectively direct the network flow records between theplurality of feature extraction processing sub-units 720 by determiningthat the network flow record 715 has a set of characteristicscorresponding to the selected feature extraction processing sub-unit720.

Each conditional operator 715 may include one or more pre-definedconditions that correspond to the downstream extractor sub-units 720.The conditional operator 715 may examine the received network flowrecords to determine whether they satisfy the conditions. Based theexamination of the network flow records, the conditional operator 715can determine the extractor sub-unit 720 to which that data can betransmitted. For example, the conditional operator 715 may route databetween the pair of downstream extractors 720 according to whether thereceived data 706 corresponds to an ingress or egress network IPaddress.

Each feature extraction sub-unit 720 can process its received datastream 708 to generate a stream 712 that includes the extractedfeatures. In some cases, the outgoing stream 712 may consist of a steamof extracted features. In other cases, the extracted features can beembedded with a portion or all of the received network flow records.

The outgoing stream 712 can be directed to a corresponding storageand/or analysis application. The analysis application may performoperations on the extracted features in stream 712 to detect events ofinterest. For example, the analysis application may perform anomalydetection or threat correlation for the selected data stream. Thestorage application may store the data in the outgoing stream 712 innon-volatile storage.

Configuring a data stream processing system to include the dataprocessing segments shown in FIGS. 6A, 6B and 7A in a processingsub-graph may be used to provide cyber-security monitoring for anetwork. For example, the processing segment 700 shown in FIG. 7A may beused to extract features from the enriched network flow records receivedfrom the processing segment shown in FIG. 6B that are usable bycyber-security monitoring and analysis applications to identifycyber-security events of interest, such as intrusions into the network.

Referring now to FIG. 7B, shown therein is another example of a dataprocessing segment 730 that can be used to extract data features fromnetwork flow records. A stream of parsed or enriched network flowrecords 732 can be received by the stream processing segment shown inFIG. 7B.

In some embodiments, the network flow records 732 may be transmittedusing UDP. In such embodiments, a UDP-to-pipe operator 735 can beimplemented to transmit the network flow records as a pipe-based stream734 as shown.

A filter operator 740 can be positioned to receive the network flowrecords 734. The filter operator 740 can be configured to transmitnetwork flow records 736 having a defined set of characteristics to theduplicator 745. The filter operator 740 can prevent network flow records734 that do not have the defined set of characteristics from beingtransmitted to the duplicator 745.

The filter operator 740 can be configured to select only data thatcorresponds to a pre-defined profile. The selected data can be providedto the duplicator 745 while the other data can be prevented fromreaching duplicator 745 and may be discarded. For example, the filteroperator may only pass HTTPS traffic or only SSH traffic.

The filtered network flow records 736 can be directed to the duplicatoroperator 745. The duplicator operator 745 can be configured to replicatethe received network flow records 736 and transmit a plurality ofreplicated network flow record streams 738. Each replicated stream 738may include all of the filtered network flow records received at 736.Each replicated stream 738 can be transmitted to one of the featureextraction processing sub-units 750.

As shown in FIG. 7B, a plurality of feature extraction processingsub-units 750 can be coupled to the duplicator operator 745. Eachfeature extraction processing sub-unit 750 can be configured to deriveone or more network flow characteristics from the duplicated networkflow records 738.

Each feature extraction sub-unit 750 can process its received datastream 738 to generate a stream 742 that includes the extractedfeatures. In some cases, the outgoing stream 742 may consist of a streamof extracted features. In other cases, the extracted features can beembedded with a portion or all of the received network flow records.

Each feature extraction sub-unit 750 can process its received datastream 738 in a different manner. For example, the plurality of featureextraction sub-units 750 may be configured to produce time-lapse graphsfor profiles of interest analysis. Each of the feature extractionsub-units 750 may then be configured to process the received data 738 indistinct but overlapping time periods. For example, ten featureextraction sub-units 750 may be configured to process ten minutes ofdata, but with windows that are shifted by 1 minute relative to eachother.

The outgoing stream 742 can be directed to a corresponding data storageand/or analysis application. The analysis application may performoperations on the extracted features in stream 742 to detect events ofinterest. The storage application may store the data in stream 742, e.g.to allow for later retrieval and/or analysis.

Configuring a data stream processing system to include the dataprocessing segments shown in FIGS. 6A, 6B and 7B in a processingsub-graph may be used to provide network flow monitoring and managementfor a network. For example, the processing segment 730 shown in FIG. 7Bmay be used to extract features from the enriched network flow recordsreceived from the processing segment shown in FIG. 6B that are usable bynetwork monitoring and analysis applications to identify events ofinterest, such as network traffic anomalies.

For instance, in the example given above, the plurality of time-lapsegraphs generated by the feature extraction sub-units 750 can define aprofile store for monitored network data. The network profiles from thefeature extraction sub-units 750 may be analyzed using signature-baseddetection applications or anomaly-detection based learning applicationsfor example. The output from these analysis applications can be used forcorrelation applications, to drive dashboards, or to initiate remedialnetwork actions for example.

Referring now to FIG. 7C, shown therein is a simplified example of aprocessing graph 760 in accordance with an embodiment. The processinggraph 760 shown in FIG. 7C illustrates a graphical example of how a usercan define a processing graph to be implemented by controller 500. Asmentioned above, a user may define the processing graph at a high levelusing a language such as Python. This high-level description can beconverted to a visual graph representation, for instance in a JSONformat.

The controller 500 may interpret the processing graph 760 in order toretrieve code defining the plurality of data processing sub-units andoperators to be deployed. The controller 500 may then deploy thecorresponding resources necessary to provide the graph.

The processing graph 760 is an example of a processing graph that can beconfigured to process a data stream for a plurality of downstreamstorage and/or analysis applications. The processing graph 760 caninclude a plurality of independent sub-graphs 770 a and 770 b. Each ofthe sub-graphs 770 can be coupled to different outgoing data streams.The sub-graphs 770 can be configured to perform different processingoperations based on the storage and/or analysis applications that areconnected to their outgoing data streams.

The sub-graphs 770 can be connected to a shared upstream portion 765 ofthe data processing graph 760. The upstream portion 765 can beconfigured to perform processing operations that are common for thestorage and/or analysis application downstream from both of theprocessing sub-graphs 770 a and 770 b. This may provide a morestreamlined processing system, by avoiding unnecessary duplication ofprocessing sub-units.

In the example processing graph 760, an incoming data stream can bereceived by a load-balancing operator 762. The load-balancing operator762 can be configured to distribute the data from the incoming datastream amongst a plurality of downstream data processing sub-units 764.

In the example shown, the load-balancing operator 762 provides acombined UDP-to-Pipe and load balancing operation. The load-balancingoperator 762 may thus convert a received UDP stream to an outgoingpipe-based data stream transmitted to the data processing sub-units 764.

The data processing sub-units 764 can be configured as parser sub-units.Each of the parser sub-units 764 can be coupled to a correspondingdownstream enrichment sub-processing unit 766. The parser sub-units 764and enrichment sub-units 766 may operate as described herein above. Byproviding a plurality of parser sub-units 764 (as well as correspondingenrichment sub-units 766) in parallel, the incoming data stream can beprocessed at a much faster rate.

Each of the enrichment data processing sub-units 766 can be connected toa replicator operator 768. Each replicator operator can be connected toboth a matching sub-unit 772 in data processing sub-graph 770 a and acombiner operator 776 in data processing sub-graph 770 b. The replicatoroperators 768 allow the output from the upstream data processingsub-unit 765 to be distributed amongst a plurality of downstream dataprocessing sub-graphs, where more specialized processing operations(i.e. those more specific to the downstream storage and/or evaluationapplications) can be performed to accommodate different storage,analysis or monitoring applications.

In data processing sub-graph 770 a, each enriched data stream can beprovided to a separate match sub-unit 772. The match sub-units 772 canperform matching operations on the received data stream, and the outputfrom the matching sub-units 772 can be directed to a combiner operator774. The combiner operator 774 can be used to direct the matched datastreams to a data store for further analysis.

In data processing sub-graph 770 b, all of the enriched data streams canbe provided to combiner operators 776. The combiner operators 776 can beused to combine the enriched data streams into a single data stream.This combined data stream can then be directed to a replicator operator778. The replicator operator 778 may then transmit multiple copies ofthe combined data stream to a plurality of extraction sub-units 782.This may ensure that each extraction sub-unit 782 analyzes the same datastream. The output from the extraction sub-units 782 can be provided todata analysis applications for further analysis and/or to data storageapplications for storage in non-volatile memory.

Referring now to FIG. 8, shown therein is an example of a dataprocessing sub-graph 800. Data processing sub-graph 800 is an example ofa data processing sub-graph that can be configured to provide datastreams to a plurality of extraction sub-units 890.

In FIG. 8, a plurality of data processing segments 850 can be arrangedin parallel. Each of the data processing segments 850 can include thesame sequence of input operator, data processing sub-units and outputoperator. For instance, the data processing sub-units may include a pipeswitch operator, enrichment processing sub-unit, and connector operatoras shown in the data processing segment of FIG. 6B.

The data processing segments 850 can be connected to a plurality ofcombiner operators 860. The combiner operators 860 can be configured tocombine the enriched data streams from two or more enrichment dataprocessing segments 850. For instance, the combiner operators 860 may beconfigured to receive UDP streams from each segment 850 and combine theminto a single stream.

The combined streams can then be provided to conditional operators 870.The conditional operators 870 can distribute the received streams into aplurality of outgoing streams by identifying the corresponding featureextraction sub-unit 890 for the given data block. For instance, theconditional operator 870 may use a hash function to identify the featureextraction sub-units 890 corresponding to each block of data. Theconditional operator 870, for each parallel segment, can then direct thedata to the sub-graph for the appropriate feature extraction sub-unit890.

Each feature extraction sub-unit 890 can be coupled to an upstreamcombiner operator 880. The combiner operator 880 can be configured toreceive data streams from some or all of the conditional operators 870,so that the data processed by each segment can be routed to each featureextraction sub-unit 890. The combiner operators 880 can then provide asingle combined stream to each feature extraction sub-unit 890. Thecombined stream can include data from all the processing segments,having been routed based on the conditions defined in the conditionaloperators 870 that operate in parallel. This may provide a highthroughput architecture for routing data to a large number of differentfeature extraction sub-units 890.

Referring now to FIGS. 9 and 10, shown therein are examples of dataanalysis sub-graphs. The data analysis sub-graphs shown in FIGS. 9 and10 are examples of data routing and analysis systems that may be used toanalyze processed data received from a data stream processing system,such as systems 105, 200 a and 200 b.

Various event detection applications may be used with embodiments of thedata stream processing systems described herein. For instance,threat-intelligence detection applications may be used to providereal-time correlation against threats reported by commercial andopen-source feeds, Host-Based detections, and internal Honey-pot baseddetections, DPI, etc. Signature-based detection applications may be usedto profile every IP address seen in the network, extract dozens offeatures per address, use graph analytics, and identify and detect inreal-time known/targeted threat patterns. In other examples, unknownanomalous behaviors may be identified using unsupervised and/orsemi-supervised machine learning-techniques that profile IP addressesusing time-lapsed moving graphs to learn normal behavior and identifyanomalous behavior.

FIG. 9 shows an example of an analysis system 900 that may be used toprovide cyber-security analysis functions. For example, the analysissystem 900 may be coupled downstream from a data stream processingsystem that incorporates the data processing segments shown in FIGS. 6A,6B and 7A. Analysis system 900 can be configured to analyze an incomingdata stream to detect threats using a correlator.

Analysis system 900 can receive an incoming data stream 901. Data stream901 is a processed data stream that has been suitably filtered and/orenriched using a data stream processing system such as those describedherein. For instance, data stream 901 may include data featuresextracted using feature extraction sub-units.

A data processing segment comprising a pipe switch operator 910, matchengine sub-unit 920, and pipe-to-UDP operator 930 can performcorrelation operations on the received data stream 901. The pipe switchoperator 910 may facilitate updating the match engine 920 on the fly.

For example, the match engine sub-unit 920 may compare the received datastream 901 against known threats identified using open source orcommercial feeds, or from other threat detection applications. Forinstance, the match sub-unit 920 may compare the IP addresses of data inthe received data stream 901 against IP addresses of known threats. Thematch sub-unit 920 may also use various other matching templates, suchas signature-based templates or templates from anomaly-detectionapplications for example.

A stream 931 including identified threats can be routed to a match store950 using a connector operator 940. The match store 950 may be used tostore a list of identified threats and associated data. The match store950 may also provide feedback to an administrator user to assist inidentifying and prioritizing detected threats. In some cases, the streamof detector threats 931 may be directed to automated alerting orresponse sub-units that may generate alerts or trigger remedial actionin response to the detected threats.

Referring now to FIG. 10, shown therein is an example of an analysissystem 1000. Analysis system 1000 is another example of an analysissystem that may be used to detect cyber-security threats and otherevents of interest. The analysis system 1000 can receive processed datastreams 1001 from the output of a data stream processing system such assystems 105, 200 a and 200 b.

Analysis system 1000 is an example of a system configured to detectthreats using signature-based applications or anomaly detectionapplications. The data streams 1001 have been filtered based on profilesof interest and processed using multiple feature extraction sub-units(e.g. using the data processing segment 730 shown in FIG. 7B). The datastreams output from the feature extraction sub-units can be combinedinto a single analysis data stream 1011 using a combiner operator 1010.The combined analysis data stream 1011 can be directed to a profilestore 1020.

A detection application 1030 can process information from the profilestore 1020 to detect threats. For example, the detection application1030 may provide signature-based detection (e.g., decision-trees basedon extracted features or ratios of extracted features) to identify newthreats. In another example, the detection application 1030 may useanomaly detection based on analytics or machine learning to identify newthreats. The outputs from one or more such detection applications 1030may then be used as an input feed to a correlation engine such as matchunit 920, as inputs to a user dashboard, or in triggering actions andresponses to mitigate new threats.

Embodiments of the data stream processing systems, method and computerprogram products described herein may facilitate ingestion andprocessing of large volumes of data. Data processing sub-units can beinterconnected using a plurality of operators to ensure that receiveddata streams are suitably processed for various storage, analysis andmonitoring applications.

Data processing sub-units can be configured to perform processingoperations—such as parsing, filtering, enriching, transforming (e.g.compressing, expanding), extracting, analyzing, storing, matching—on areceived data stream and output a modified data based on the operationsperformed. The data processing sub-units may be implemented using highperformance languages such as C/C++.

The operators can route data between the plurality of data processingsub-units and provide required connectivity between different dataprocessing sub-units. The operators can route data using variouscriteria, such as load balancing, conditional routing, switching,replication, joining etc. The operators may also modify thecommunication protocols used to transmit data to provide low latency aswell as increased flexibility in connections between the data processingsub-units.

The plurality of data processing sub-units and operators can be arrangedin a feedforward data processing graph. The data processing graph mayinclude parallel segments of data processing sub-units and operators toincrease system throughput and performance. The operators can also beconfigured to dynamically adjust the connections between data processingsub-units and data processing segments to facilitate resource scaling asrequired. This may reduce, or prevent, data throttling through the datastream processing system.

The inventors have found that embodiments of the systems, methods andcomputer program products described herein can process events at leastten times more efficiently than existing “best approaches” such asSpark, Hadoop, and ELK using the same computational resources.Embodiments described herein may be implemented to process more than150,000 events per second and upwards of millions of events per second.This high event throughput may allow system monitoring with little or nosubsampling in large networks. This can facilitate real-time analysisand detection with greater accuracy as all, or almost all, themonitoring data for the network can be processed and analyzed.

While the above description describes features of example embodiments,it will be appreciated that some features and/or functions of thedescribed embodiments are susceptible to modification without departingfrom the spirit and principles of operation of the describedembodiments. For example, the various characteristics which aredescribed by means of the represented embodiments or examples may beselectively combined with each other. Accordingly, what has beendescribed above is intended to be illustrative of the claimed conceptand non-limiting. It will be understood by persons skilled in the artthat other variants and modifications may be made without departing fromthe scope of the invention as defined in the claims appended hereto. Thescope of the claims should not be limited by the preferred embodimentsand examples, but should be given the broadest interpretation consistentwith the description as a whole.

We claim:
 1. A system for processing network flow monitoring datacomprising: a data collection input coupled to at least one incomingdata stream of network monitoring messages; and a data processingsegment comprising: a load balancing operator coupled to the datacollection input, the load balancing operator operable to distribute thenetwork monitoring messages, for processing, amongst a plurality ofdownstream collector processing sub-units coupled to the load balancingoperator, wherein the load balancing operator is configured todistribute the network monitoring messages based on a performance of oneor more of the plurality of collector processing sub-units; a dataprocessing sequence that includes: the plurality of collector processingsub-units, wherein the collector processing sub-units are operable togenerate network flow records from the received network monitoringmessages, wherein the network flow records are generated in a firstformat and the collector processing sub-units are configured to generatethe network flow records by augmenting the received network monitoringmessages with additional information; and a plurality of parsersub-units coupled to the plurality of collector processing sub-units,each parser sub-unit operable to receive the network flow records fromone of the collector processing sub-units and to generate parsed networkflow records by converting the received network flow records from thefirst format to a second format; a combiner operator coupled to theplurality of parser sub-units, wherein the combiner operator is operableto combine the parsed network flow records from the plurality of parsersub-units into a synchronous output data stream; and a data processingsub-unit configured to generate a profile store utilizing graphs,wherein the profiles from the profile store may be analyzed by eitherdetection applications or machine learning, wherein the result of theanalysis may be utilized for at least one of correlating applications,interacting with dashboards, or facilitating remedial network actions;wherein the plurality of parser sub-units are directly connected to theplurality of collector processing sub-units within the data processingsequence; the plurality of parser sub-units are directly connected tothe combiner operator; the plurality of collector processing sub-unitsare directly connected to the load balancing operator; and data flowsthrough the data processing segment sequentially from the load balancingoperator to the plurality of collector processing sub-units to theplurality of parser sub-units to the combiner operator.
 2. The system ofclaim 1, wherein the load balancer operator is configured to transmitthe network monitoring messages to the plurality of collector processingsub-units using the User Datagram Protocol (UDP).
 3. The system of claim1, wherein each parser sub-unit is configured to generate the parsednetwork flow records in a CSV file format.
 4. The system of claim 1,wherein each parser sub-unit is configured to transmit the parsednetwork flow records using the User Datagram Protocol (UDP).
 5. Thesystem of claim 1, further comprising: at least one stream enrichmentprocessing sub-unit coupled downstream from the combiner operator, eachstream enrichment processing sub-unit operable to generate enrichednetwork flow records by inserting enrichment data into the parsednetwork flow records.
 6. The system of claim 5, further comprising: astream output operator coupled to the at least one stream enrichmentprocessing sub-unit, the stream output operator configured to output theenriched network flow records using the User Datagram Protocol (UDP). 7.The system of claim 1, further comprising: a duplicator operator coupleddownstream from the combiner operator, the duplicator operatorconfigured to duplicate the received network flow records; and aplurality of feature extraction processing sub-units coupled to theduplicator operator, each feature extraction processing sub-unitoperable to derive one or more network flow characteristics from theduplicated network flow records.
 8. The system of claim 7, furthercomprising: a plurality of conditional operators coupled between theduplicator and the plurality of feature extraction processing sub-units,wherein each conditional operator is operable to selectively direct thenetwork flow records between the plurality of feature extractionprocessing sub-units by determining that the network flow record has acharacteristic corresponding to the selected feature extractionprocessing sub-unit.
 9. The system of claim 7, further comprising: afilter operator coupled upstream from the duplicator, wherein the filteroperator is operable to transmit network flow records having a definedset of characteristics to the duplicator and to prevent network flowrecords that do not have the defined set of characteristics from beingtransmitted to the duplicator.
 10. The system of claim 1, furthercomprising at least one data output unit coupled downstream of thecombiner operator, wherein the at least one data output unit comprisesat least one of a data analysis application and a real-time storageapplication.
 11. A method of processing network flow monitoring data,the method comprising: receiving at least one incoming data stream ofnetwork monitoring messages; directing the network monitoring messagesto a data processing segment comprising a load balancing operator, adata processing sequence that includes a plurality of collectorprocessing sub-units and a plurality of parser sub-units, and a combineroperator, wherein the plurality of parser sub-units are directlyconnected to the plurality of collector processing sub-units, theplurality of parser sub-units are directly connected to the combineroperator, and the plurality of collector processing sub-units aredirectly connected to the load balancing operator such that data flowsthrough the data processing segment sequentially from the load balancingoperator to the plurality of collector processing sub-units to theplurality of parser sub-units to the combiner operator; distributing bythe load balancing operator, for processing, the network monitoringmessages amongst the plurality of collector processing sub-units basedon a performance of one or more of the plurality of collector processingsub-units; generating, by the plurality of collector processingsub-units, network flow records from the received network monitoringmessages, wherein the network flow records are generated in a firstformat and the network flow records are generated by augmenting thereceived network monitoring messages with additional information;transmitting, by the plurality of collector processing sub-units, thenetwork flow records to a plurality of parser processing sub-unitsdownstream from the collector processing sub-units; generating, by theplurality of parser processing sub-units, parsed network flow records byconverting the received network flow records from the first format to asecond format; combining, by the combiner operator, the parsed networkflow records from the plurality of parser sub-units into a synchronousoutput data stream; and generating a profile store utilizing graphs,wherein the profiles from the profile store may be analyzed by eitherdetection applications or machine learning, wherein the result of theanalysis may be utilized for at least one of correlating applications,interacting with dashboards, or facilitating remedial network actions.12. The method of claim 11, wherein the network monitoring messages aredistributed amongst the plurality of collector processing sub-unitsusing the User Datagram Protocol (UDP).
 13. The method of claim 11,wherein the parsed network flow records are generated in a CSV fileformat.
 14. The method of claim 11, wherein the parsed network flowrecords are output from the parser processing sub-units using the UserDatagram Protocol (UDP).
 15. The method of claim 11, further comprisinggenerating enriched network flow records by inserting enrichment datainto the parsed network flow records.
 16. The method of claim 15,further comprising transmitting the enriched network flow records usingthe User Datagram Protocol (UDP).
 17. The method of claim 11, furthercomprising: duplicating the network flow records; transmitting theduplicated network flow records to a plurality of feature extractionprocessing sub-units; and deriving, by each feature extractionprocessing sub-unit, one or more network flow characteristics from theduplicated network flow records.
 18. The method of claim 17, whereintransmitting the duplicated network flow records to the plurality offeature extraction processing sub-units comprises selectively directingeach network flow record between the plurality of feature extractionprocessing sub-units by determining that the network flow record has acharacteristic corresponding to the selected feature extractionprocessing sub-unit.
 19. The method of claim 17, further comprising:filtering the network flow records prior to duplication, wherein networkflow records having a defined set of characteristics are duplicated andnetwork flow records that do not have the defined set of characteristicsare prevented from passing through the filter.
 20. The method of claim11, further comprising routing the output data stream to at least onedata output unit, wherein the at least one data output unit comprises atleast one of a data analysis application and a real-time storageapplication.
 21. A computer program product comprising a non-transitorycomputer-readable medium having computer-executable instructions storedtherein, the computer-executable instructions being executable by aprocessor to configure the processor to perform a method of processingnetwork flow monitoring data, wherein the method comprises: receiving atleast one incoming data stream of network monitoring messages; directingthe network monitoring messages to a data processing segment comprisinga load balancing operator, a data processing sequence that includes aplurality of collector processing sub-units and a plurality of parsersub-units, and a combiner operator, wherein the plurality of parsersub-units are directly connected to the plurality of collectorprocessing sub-units, the plurality of parser sub-units are directlyconnected to the combiner operator, and the plurality of collectorprocessing sub-units are directly connected to the load balancingoperator such that data flows through the data processing segmentsequentially from the load balancing operator to the plurality ofcollector processing sub-units to the plurality of parser sub-units tothe combiner operator; distributing by the load balancing operator, forprocessing, the network monitoring messages amongst the plurality ofcollector processing sub-units based on a performance of one or more ofthe plurality of collector processing sub-units; generating, by theplurality of collector processing sub-units, network flow records fromthe received network monitoring messages, wherein the network flowrecords are generated in a first format and the network flow records aregenerated by augmenting the received network monitoring messages withadditional information; transmitting, by the plurality of collectorprocessing sub-units, the network flow records to a plurality of parserprocessing sub-units downstream from the collector processing sub-units;generating, by the plurality of parser processing sub-units, parsednetwork flow records by converting the received network flow recordsfrom the first format to a second format; combining, by the combineroperator, the parsed network flow records from the plurality of parsersub-units into a synchronous output data stream; and generating aprofile store utilizing graphs, wherein the profiles from the profilestore may be analyzed by either detection applications or machinelearning, wherein the result of the analysis may be utilized for atleast one of correlating applications, interacting with dashboards, orfacilitating remedial network actions.